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Foreword 


The  Capability  Maturity  Model®  Integration  (CMMI)  product  suite  is  widely  known  and 
employed  across  the  commercial  and  defense  product  development  community  within  the  United 
States  and  a  number  of  other  countries.  As  stewards  of  CMMI,  the  Software  Engineering  Institute 
(SEI)  and  the  National  Defense  Industrial  Association  (NDIA)  want  to  ensure  there  is  adequate 
guidance  for  organizations  involved  in  acquisition  to  leverage  CMMI’s  disciplined  practices. 

We  developed  this  guidebook  for  acquisition  office  personnel  who  are  responsible  for  planning 
the  acquisition  strategy  and  preparing  information  associated  with  a  request  for  proposal  for 
systems  acquisition.  This  guidebook  explains  CMMI  terminology  and  the  meaning  of  CMMI 
ratings.  The  intent  is  to  promote  effective  communication  with  supplier  organizations  and  avoid 
misuse  and  misinterpretation  of  the  model. 

Contributors  to  the  original  version  of  this  guidebook  include  Dr.  Karen  Richter,  Institute  for 
Defense  Analyses;  Mr.  Joseph  P.  Elm,  Mr.  Jeffery  L.  Dutton,  Dr.  Gene  Miluk,  Mr.  Mike  Phillips, 
and  Mr.  Joe  Wickless  from  the  Software  Engineering  Institute;  Ms.  Linda  M.  Rosa,  The  MITRE 
Corporation;  Mr.  Brian  P.  Gallagher  and  Mr.  Hal  Wilson  from  Northrop  Grumman  Information 
Systems;  and  Mr.  Lawrence  T.  Osiecki,  U.S.  Army  ARDEC.  Key  reviewers  of  this  guide  include 
the  members  of  the  CMMI  Steering  Group. 

Finally,  our  special  thanks  are  reserved  for  Mr.  Lawrence  T.  Osiecki,  U.S.  Army  ARDEC,  Mr. 
John  Scibilia,  U.S.  Army  ARDEC,  and  Mr.  Mike  Phillips,  Software  Engineering  Institute  for 
aligning  this  text  with  CMMI -DEV  version  1.3  modifications.  Their  effort  made  this  updated 
guidebook  a  reality. 


Paul  D.  Nielsen 

Director  and  Chief  Executive  Officer 
Software  Engineering  Institute  Carnegie 
Mellon  University 


Robert  C.  Rassa,  Raytheon 
Chairman 

Systems  Engineering  Division 
National  Defense  Industrial  Association 
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Executive  Summary 


Capability  Maturity  Model  Integration  (CMMI)  was  developed  as  an  integrated  approach  to 
internal  process  improvement  in  a  broad  range  of  organizations.  CMMI  for  Development  (CMMI- 
DEV),  Version  1.3,  consists  of  a  collection  of  best  practices  that  address  product  development  and 
maintenance  activities  that  cover  the  product  lifecycle,  from  conception  through  delivery  and 
sustainment. 

This  guidebook  is  designed  to  help  an  acquirer  benefit  from  a  supplier’s  use  of  CMMI-DEV  while 
helping  the  acquiring  organization  avoid  the  pitfalls  associated  with  unrealistic  expectations.  This 
guidebook  discusses  how  to 

•  recognize  methods  that  leverage  a  supplier’s  process  improvement  initiatives 

•  determine  which  process  areas  (PAs)  are  critical  to  the  success  of  the  program* 

•  request,  understand,  interpret,  and  use  the  results  of  a  supplier’s  CMMI  appraisals 

•  interpret  a  supplier’s  claims  of  achieving  a  CMMI  rating 

•  gather  and  interpret  information  for  effectively  monitoring  supplier  processes  throughout  the 
execution  of  an  acquisition  program 

•  understand  the  significance  of  the  improvements  to  the  CMMI  Product  Suite  in  vl.3 

This  guidebook  also  describes  CMMI  fundamentals  that  acquirers  must  understand  to  effectively 
use  information  obtained  from  a  supplier’s  CMMI-DEV  efforts  on  development  programs.  It 
includes  explanations  of  capability  and  maturity  levels  and  the  differences  between  the  two 
CMMI  representations — continuous  and  staged.  The  terms  and  concepts  explained  are  ones 
acquirers  may  encounter  in  proposals  and  in  everyday  dealings  with  suppliers.  This  guidebook 
intends  to  demystify  these  terms  and  concepts. 

Acquirers  and  users  of  CMMI-DEV  should  be  cautioned  that  maturity  level  (ML)  ratings 
alone  do  not  guarantee  program  success.  CMMI-DEV  is  an  important  tool  for  program 
managers  to  assess  the  completeness  of  their  development  practices  and  to  guide  internal  process 
improvement.  This  guidebook  demonstrates  how  to  apply  this  tool  and  how  an  acquiring 
organization  can  benefit  from  this  application. 

For  more  information  about  the  CMMI  Product  Suite,  refer  to  the  CMMI  website  at  the  following 
URL:  <http://www.sei.cmu.edu/cmmi/>. 


This  report  uses  the  term  program  in  piece  of  the  CMMI  modei  standard  term  project. 
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Abstract 


This  guidebook  is  designed  to  help  acquisition  organizations  formulate  questions  for  their 
suppliers  related  to  using  CMMI.  It  also  helps  organizations  identify  and  evaluate  risks  in  supplier 
processes  by  explaining  how  to  gather  and  interpret  supplier  data  throughout  the  acquisition  and 
then  to  effectively  monitor  supplier  processes  after  contract  award. 

This  guidebook  helps  clarify  what  high  capability  and  maturity  level  ratings  signify  in  a 
development  program  and  describes  how  acquirers  can  apply  methods  that  leverage  a  supplier’s 
process  improvement  initiatives;  request,  understand,  interpret,  and  use  supplier  appraisal  results; 
and  interpret  suppliers’  claims  of  achieving  a  CMMI  rating. 
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1  Introduction 


This  guidebook  helps  acquisition  organizations^  formulate  questions  for  their  suppliers  related  to 
CMMI.  It  also  helps  organizations  interpret  responses  to  identify  and  evaluate  risks  for  a  given 
supplier.  The  guidebook  provides  approaches  for  acquisition  organizations  to  gather  and  interpret 
supplier  data  throughout  the  acquisition  process  to  effectively  monitor  supplier  processes  that  are 
part  of  the  acquisition  program  after  contract  award.  This  guidebook  deals  exclusively  with 
supplier  processes.  It  does  not  address  issues  of  process  application,  process  capability,  or 
organizational  maturity  within  the  acquisition  organization. 
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Figure  1:  Guidebook  Focus — Understanding  Suppiier  Processes 


CMMI-DEV  contains  process  requirements  based  on  industry  best  practices  that  can  be  used  to 
guide  organizations  working  to  improve  their  development  processes.  Organizations  that 
concentrate  on  process  improvement  demonstrate  reductions  in  delivered  defects,  improved  speed 
of  development,  and  fewer  post-delivery  failures.  Given  those  results,  organizations  that 
implement  CMMI -based  process  improvement  may  provide  a  lower  risk  in  development  over 
those  that  do  not. 


CMMI-DEV  provides  a  reference  against  which  organizations  can  be  appraised  using  formal 
appraisal  techniques.  Using  CMMI  appraisal  data  for  evaluation  in  acquisitions  can  be  an 
effective  means  to  identify  and  evaluate  risk  within  the  development  portion  of  an  acquisition. 
CMMI  appraisal  data,  obtained  using  the  Standard  CMMI  Appraisal  Method  for  Process 
Improvement  (SCAMPI^^),  identify  and  evaluate  development  processes  that  organizations 
defined  and  utilized  in  the  execution  of  their  programs.  Thus,  this  data  can  be  used  to  help  identify 
the  completeness  of  the  supplier’s  processes  and  evidence  of  the  use  of  those  processes.  This 
allows  assessment  of  the  degree  of  risk  related  to  a  supplier’s  organizational  processes  used  for 
product  development. 


An  acquisition  organization  may  be  a  government  acquirer  (e.g.,  program  management  office  [PMO],  joint 
program  office  [JPO]),  a  government  contractor  serving  as  an  acquirer  for  the  government,  or  the  acquisition 
organization  within  a  commercial  enterprise.  This  guidebook  is  written  for  all  of  these  audiences,  and  the  term 
acquirer  represents  all  of  them. 
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CMMI-DEV  and  its  appraisal  method  provide  an  approach  for  determining  how  well  the 
appraised  organization  has  employed  its  processes  in  the  programs  evaluated  as  part  of  the 
appraisal.  This  approach  can  indicate  the  degree  of  risk  (or  lack  of  risk)  that  these  same  processes 
might  contribute  when  implemented  on  a  new  program.  This  indication  is  only  valid,  however,  if 
the  supplier  actually  uses  the  appraised  processes  on  the  program  at  hand.  Implementation  of 
these  processes  should  begin  before  contract  award  (in  the  capture  phase),  and  continue  in  earnest 
post  award  in  order  to  reap  the  benefits.  This  guidebook  provides  guidance  to  help  ensure  that  this 
consistent  use  of  appraised  processes  by  the  supplier  happens. 

Sections  1 . 1  and  1 .2  provide  foundational  material  on  CMMI-DEV  and  CMMI  appraisals. 

Readers  familiar  with  CMMI-DEV  and  the  appraisal  method  may  skip  these  sections.  Section  1.3 
includes  information  to  help  acquirers  interpret  the  results  of  CMMI  appraisals. 

1.1  CMMI-DEV  Fundamentals 

CMMI-DEV  consists  of  a  set  of  process  requirements  based  on  industry  best  practices  that  are 
organized  into  22  different  process  areas  across  four  categories,  as  shown  in  Table  1.  (The  process 
areas  related  to  Project  Management,  Engineering,  and  Support  are  discussed  in  Chapter  2.)  The 
Process  Management  process  areas  are  used  at  the  organization  level  to  define,  deploy,  and 
improve  processes  across  the  organization.  The  process  areas  are  important  because  they  play  a 
large  part  in  how  effectively  the  organization  deploys  its  processes  in  a  new  program. 

Table  1:  CMMI-DEV  Process  Areas 


Process  Category 

Project  Management 

Engineering 

Support 

Process  Management 

Project  Planning  (PP) 

Requirements 

Development  (RD) 

Configuration 

Management  (CM) 

Qrganizational  Process 
Focus  (QPF) 

Project  Monitoring  and 
Control  (PMC) 

Technical  Solution  (TS) 

Process  and  Product 

Quality  Assurance  (PPQA) 

Qrganizational  Process 
Definition  (QPD) 

Supplier  Agreement 
Management  (SAM) 

Product  Integration  (PI) 

Measurement  and 

Analysis  (MA) 

Qrganizational  Training 
(OT) 

Integrated  Project 
Management  (IPM) 

Verification  (VER) 

Decision  Analysis  and 
Resolution  (DAR) 

Qrganizational  Process 
Performance  (QPP) 

Risk  Management  (RSKM) 

Validation  (VAL) 

Causal  Analysis  and 
Resolution  (CAR) 

Qrganizational 

Performance  Management 
(QPM) 

Quantitative  Project 
Management  (QPM) 

Requirements 

Management  (REQM) 

CMMI-DEV  does  not  dictate  specific  processes,  but  rather  defines  the  best  practices  that  suppliers 
incorporate  into  their  development  processes.  The  degree  to  which  an  organization’s  development 
processes  conform  to  CMMI-DEV  is  measured  using  an  appraisal  on  a  representative  sample  of 
programs  in  the  appraised  organization.  Many  organizations  use  appraisals  as  a  means  of 
assessing  their  process  capabilities  and  guiding  process  improvement  activities. 
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An  appraisal  can  result  in  a  capability  level  profile  across  a  number  of  process  areas  or  a  maturity 
level  rating  for  the  organization,  depending  on  the  model  representation  used.  CMMI-DEV  has 
two  representations  that  are  used  for  appraisals — continuous  and  staged — that  lead  to  capability 
level  and  maturity  level  ratings,  respectively.  The  staged  representation  predefines  the  appraisal 
structure  for  each  grouping  of  process  areas,  while  an  appraisal  using  the  continuous 
representation  appraises  each  selected  process  area  independently.  Organizations  may  choose  one 
representation  over  the  other  for  a  variety  of  reasons,  including  the  current  state  of  ongoing 
improvement  initiatives,  the  supplier’s  historical  familiarity  with  a  particular  approach,  and 
perceived  business  needs  and  objectives. 

In  general,  an  organization’s  progress  in  defining  and  improving  its  processes  is  measured  using 
numerical  levels  of  capability  or  maturity.  Higher  levels  indicate  increasing  degrees  of 
sophistication  and  institutionalization  of  the  process  improvement  efforts  in  the  organization. 

The  continuous  representation  of  CMMI-DEV  measures  process  capability  within  each  process 
area  in  an  ordered  grouping  of  four  capability  levels  represented  by  the  numbers  0-3.  It  allows  an 
organization  to  choose  which  process  areas  to  appraise  based  on  its  business  objectives  and 
process  improvement  goals.  An  appraisal  using  the  continuous  representation  consists  of  a 
capability  level  profile  showing  the  capability  level  achieved  within  each  chosen  process  area 
interpreted  as  follows: 

•  Capability  level  0  indicates  that  the  process  is  either  not  performed  or  only  partially 
performed. 

•  Capability  level  1  indicates  that  the  process  is  performed  to  the  extent  that  it  meets  the  goals 
of  the  process  and  produces  the  necessary  products. 

•  Capability  level  2  indicates  that  the  process  is  managed  in  accordance  with  a  policy. 

•  Capability  level  3  indicates  that  the  process  is  tailored  from  the  organization’s  set  of  standard 
processes. 

The  staged  representation  specifies  sets  of  process  areas  in  an  ordered  grouping  of  five  maturity 
levels  and  predefines  which  process  areas  must  be  successfully  appraised  to  achieve  a  maturity 
level  rating.  An  appraisal  for  the  staged  representation  results  in  a  maturity  level  rating, 
interpreted  as  follows: 

•  Maturity  level  1  indicates  that  processes  are  usually  ad  hoc. 

•  Maturity  level  2  indicates  that  the  organization  has  achieved  capability  level  2  for  each 
process  area  at  maturity  level  2.  Maturity  level  2  is  primarily  demonstrated  at  the  program 
level  and  focuses  on  the  Project  Management  and  Support  process  areas. 

•  Maturity  level  3  indicates  that  the  organization  has  achieved  capability  level  3  for  all  maturity 
level  2  and  maturity  level  3  process  areas,  which  include  the  Engineering  and  Process 
Management  process  areas.  It  also  indicates  that  the  organization  has  adopted  a  process  focus 
at  an  organizational  level  and  has  a  set  of  standard  processes  that  can  be  tailored  for  specific 
programs. 
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•  Maturity  level  4  indicates  that  the  organization  has  demonstrated  that  it  quantitatively 
manages  selected  processes  (program  and  organizational  activities  that  are  deemed  important 
and  are  consistent  with  business  objectives)  and  that  it  has  achieved  capability  level  3  on  all 
maturity  level  2,  3,  and  4  process  areas. 

•  Maturity  level  5  indicates  that  the  organization  has  demonstrated  that  it  has  used  its 
measurement  data  to  improve  and  optimize  selected  processes  and  subprocesses,  and  that  it 
has  achieved  capability  level  3  on  all  process  areas  in  the  model. 

Organizations  that  use  the  continuous  representation  can  convert  their  appraised  process  area 
results  into  an  organizational  maturity  level  rating  using  equivalent  staging.  Figure  2,  adapted 
from  CMMI-DEV,  illustrates  the  grouping  of  process  areas  into  the  maturity  levels  of  the  staged 
representation,  the  rules  for  equivalent  staging,  and  what  constitutes  high  maturity. 

When  practitioners  of  CMMI-DEV  refer  to  high  maturity,  they  are  referring  to  behaviors 
associated  with  organizations  performing  at  maturity  level  4  or  5.  Such  behavior  entails 
quantitatively  managing  and  optimizing  a  limited  number  of  processes  or  subprocesses  that  are 
required  in  achieving  maturity  level  3  and  that  contain  the  complete  set  of  development  process 
areas.  Maturity  levels  4  and  5  focus  on  effectively  managing  and  improving  the  basic  set  of 
development  processes.  They  do  not  contain  processes  that  directly  apply  to  development. 

The  grouping  of  process  areas  into  maturity  levels  is  no  indication  of  their  relative  importance.  It 
merely  illustrates  a  path  or  sequence  for  process  improvement. 
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Figure  2:  Illustration  of  Maturity  Level  Groupings,  Equivalent  Staging,  and  High  Maturity  CMMI 
Appraisals 

Three  types  (generally  referred  to  as  classes)  of  the  SCAMPI  appraisal  method  are  defined: 
SCAMPI  Class  A,  Class  B,  and  Class  C.  Each  class  has  a  different  purpose,  but  all  are  capable  of 
identifying  process-related  strengths,  weaknesses,  and  risks.  The  appraisal  team  examines  one  or 
more  programs  within  the  organization  for  compliance  to  the  portions  of  the  CMMI  model 
specified  in  the  appraisal  scope.  The  appraisal  process  generally  involves  the  review  of  process 
documentation  (e.g.,  policies  and  procedures),  examination  of  process  artifacts  (e.g.,  work 
products),  and  interviews  with  staff  to  ascertain  the  degree  of  process  deployment  within  the 
organization.  Irrespective  of  the  representation  of  the  model  the  organization  selected,  the 
SCAMPI  appraisal  method  applies. 
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•  SCAMPI  Class  A  is  the  only  formal  appraisal  type  available  for  official  capability  or 
maturity  level  ratings.  Suppliers  that  claim  a  maturity  or  capability  level  rating  can  do  so  only 
as  a  result  of  a  SCAMPI  Class  A  appraisal. 

•  SCAMPI  Class  B  generally  is  used  for  internal  process  improvement,  as  it  does  not  result  in 
a  maturity  level  or  capability  level  rating.  The  SCAMPI  Class  B  appraisal  identifies  gaps  in 
process  implementation  and  is  commonly  referred  to  as  a  gap  analysis. 

•  SCAMPI  Class  C  (commonly  called  a  “quick  look”)  is  less  comprehensive,  but  more  flexible 
than  the  SCAMPI  Class  B,  and  can  be  performed  with  a  small  team.  There  are  fewer 
evidentiary  requirements  for  a  SCAMPI  Class  C  than  for  a  SCAMPI  Class  B  appraisal.  Often, 
a  SCAMPI-C  appraisal  is  a  document  or  artifact  review  without  staff  member  interviews.  It 
primarily  identifies  gaps  in  process  implementation. 

Conducting  a  SCAMPI  Class  A  appraisal  requires  the  participation  of  a  certified  SCAMPI  Lead 
Appraiser®'^  authorized  by  the  Software  Engineering  Institute  (SEI).  The  lead  appraiser  for  a 
SCAMPI  Class  A  must  be  independent  of  the  organizational  unit  being  appraised.  Qualified  lead 
appraisers  are  listed  in  the  SEI  Partner  Network  Directory  [SEI  1].  Conducting  a  SCAMPI  Class 
B  or  SCAMPI  Class  C  appraisal  requires  a  trained  and  qualified  appraisal  team  leader  authorized 
by  the  SEI.  The  team  leader  for  these  appraisals  is  not  required  to  be  from  an  external 
organization.  Qualified  SCAMPI  Class  B  and  C  Team  Leaders  also  are  listed  in  the  SEI  Partner 
Network  Directory.  Both  SCAMPI  Lead  Appraisers  and  Team  Leaders  will  be  referred  to 
throughout  this  document  as  appraisal  leads. 

Appraisal  team  members  that  completed  the  training  requirements  delineated  by  the  SEI  may  be 
drawn  from  within  the  appraised  organization  or  external  organizations.  Generally,  an 
organization  conducting  a  SCAMPI  Class  A  appraisal  may  include  more  external  than  internal 
staff  members  to  clearly  demonstrate  appraisal  independence  to  bolster  any  maturity  or  capability 
level  claim. 

The  results  of  SCAMPI  Class  A  appraisals  are  documented  in  an  Appraisal  Disclosure  Statement 
(ADS)  and  an  appraisal  findings  document.  The  lead  appraiser  provides  these  documents  to  the 
organization  (i.e.,  the  appraisal  sponsor — ^normally  a  senior  manager  of  the  organization  being 
appraised)  and  to  the  SEI  for  validation.  Until  the  SEI  validates  appraisal  results,  any  claims  made 
by  suppliers  about  levels  are  not  valid. 

For  a  SCAMPI  A  appraisal,  many  organizations  also  permit  the  appraisal  results  to  be  publicly 
posted  on  the  SEI  Published  Appraisal  Report  Site  (PARS)  [SEI  2].  The  acquirer  should  always 
check  the  PARS  to  validate  a  claim,  since  only  validated  results  are  posted  on  PARS.  Since  not  all 
organizations  post  results  on  PARS,  nor  is  the  information  there  a  complete  Appraisal  Disclosure 
Statement,  acquirers  may  prefer  to  request  a  copy  of  the  most  recent  ADS  from  the  appraised 
organization.  (See  “Consider  Recent  Appraisal  Results”  on  page  46  in  Appendix  D  for 
implementation  details.)  Additional  information  about  the  ADS  is  provided  in  Section  3.6. 

The  SCAMPI  Class  A  appraisal  is  described  in  detail  in  the  Standard  CMMI  Appraisal  Method 
for  Process  Improvement  (SCAMPI  A,  Version  1.3:  Method  Definition  Document  [SEI  2011]. 
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SCAMPI  Class  B  and  Class  C  appraisals  are  described  in  the  Handbook  for  Conducting  Standard 
CMMI  Appraisal  Method for  Process  Improvement  (SCAMPI)  Class  B  and  C  Appraisals  [Hayes 
2005]. 

1.2  Interpreting  CMMI  Ratings 

Various  claims  made  about  capability  models  and  maturity  level  ratings  over  the  years  have  led  to 
some  misperceptions.  Organizations  that  attain  a  CMMI  maturity  level  rating  of  3  or  higher 
should  possess  documented  organizational  standards,  defined  processes,  and  procedures  that  are 
demonstrated  to  be  repeatable. 

There  are  documented  results,  including  those  provided  by  a  government  development 
organization,  that  show  measured  benefits  for  use  of  processes  that  adhere  to  the  CMMI 
development  model.  The  same  results  show  that  those  organizations  with  a  history  of  continuous 
improvement  and  a  commitment  to  continued  utilization  of  their  development  practices  can 
produce  high  quality  results.  That  is  the  value  of  CMMI  use.  It  is  up  to  the  acquiring  organization 
to  determine  if  the  supplier  is  such  an  organization  or  just  one  who  claims  to  be. 

While  a  maturity  level  3  supplier  should  tailor  the  organization’s  standard  processes  for  use  on  all 
new  programs,  the  acquirer  cannot  infer  that  the  processes  are  necessarily  effective  or  even 
consistently  applied  across  the  enterprise.  A  more  thorough  examination  of  this  topic,  including 
suggestions  for  the  acquirer  to  investigate  supplier’s  claims,  is  presented  in  Appendix  A.  The 
discussion  centers  on  maturity  level  ratings  rather  than  process  capability  profiles,  since  the 
majority  of  appraised  organizations  end  up  with  a  maturity  level  rating,  and  maturity  levels  are 
most  often  touted  by  suppliers.  Some  warnings  for  the  acquirer  follow: 

•  A  CMMI  rating  or  CMMI  level  is  not  a  guarantee  of  program  success. 

•  Organizations  that  have  attained  CMMI  maturity  level  ratings  do  not  necessarily  apply  those 
appraised  processes  to  a  new  program  at  program  start-up. 

•  Organizations  that  claim  CMMI  ratings  may  not  remain  committed  to  process  improvement. 

•  CMMI  level  ratings  are  based  on  a  representative  sample  of  an  organization’s  programs  and 
may  not  reflect  performance  on  all  programs 

•  Organizations  that  claim  a  high  maturity  level  rating  (level  4  and  5)  are  not  necessarily  better 
suppliers  than  a  level  3  supplier. 

A  maturity  level  rating  is  not  an  indication  of  how  the  next  program  will  perform',  it  is  only  an 
indication  of  how  the  next  program  could  perform  provided  other  critical  success  factors  remain 
the  same.  Clearly,  past  performance  only  indicates  the  potential  performance  for  the  current 
program.  To  achieve  the  anticipated  performance  on  the  current  program,  the  mature  organization 
must  instantiate  the  appraised  processes  on  that  program. 

1 .3  Overview  of  this  Guidebook 

While  this  guidebook  is  intended  for  use  by  those  without  an  extensive  background  in  CMMI, 
some  readers  may  feel  that  they  need  additional  information  on  process  capability  and  process 
improvement.  Additional  sources  of  help  include  the  Software  Engineering  Institute  (SEI),  the 
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Defense  Contract  Management  Agency  (DCMA),  and  the  various  software  engineering  centers 
(SECs)  or  software  support  activities  (SSA)  established  within  the  Services. 

If  acquirers  choose  to  capitalize  on  their  suppliers’  use  of  CMMI-DEV,  the  remainder  of  this 
guidebook  describes  approaches  for  solicitation  planning  and  execution  as  well  as  contract 
monitoring. 

•  Chapter  2  covers  identifying  and  understanding  the  critical  process  areas  of  the  program. 

•  Chapter  3  describes  approaches  for  leveraging  the  process  capabilities  of  the  supplier  by 
enabling  them  in  the  request  for  proposal  (REP). 

•  Chapter  4  describes  how  to  monitor  the  supplier’s  performance  to  ensure  that  the  identified 
process  capabilities  are  being  applied  to  the  program. 

A  detailed  discussion  about  implementing  these  approaches  is  presented  in  the  appendices.  The 
acquirer  must  determine  which  of  these  methods  are  applicable  to  its  program.  Appendix  B 
contains  a  checklist  of  the  methods  identified  in  this  document.  It  can  be  used  as  an  aid  in 
implementing  and  tracking  the  status  of  each  selected  method. 
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2  Identify  the  Critical  Process  Areas  of  the  Program 


Identifying  and  understanding  the  processes  that  are  most  critical  to  the  success  of  the  program  is 
an  important  first  step  in  determining  the  criticality  of  process-related  risks.  This  chapter 
describes  how  to  determine  the  critical  process  areas  of  the  program.  The  more  critical  a  process 
may  be  to  the  program’s  success,  the  more  importance  should  be  placed  on  the  capability  of  the 
supplier’s  process,  and  the  more  risk  may  be  incurred  if  its  process  is  not  well  defined  and 
implemented. 

Suppliers  that  establish  and  use  effective  processes,  working  as  part  of  a  mature  organization,  are 
more  likely  to  do  the  following: 

•  plan  and  execute  development  programs  predictably 

•  support  those  programs  with  an  engineering  and  management  infrastructure  that  will  enable 
them  to  succeed 

•  communicate  more  accurately  and  frequently  with  customers  and  stakeholders 

•  stay  on  schedule  and  deliver  quality  products 

Evaluating  the  supplier’s  proposed  development  processes — and  then  ensuring  that  the  supplier 
actually  applies  those  processes  to  the  program  from  the  start — are  the  acquirer’s  primary  process 
challenges.  To  meet  these  challenges,  the  acquirer  must  first  understand  the  critical  process  areas 
of  the  program. 

2.1  Estimate  Program  Dependence  on  Process  Capability 

Experience  has  shown  that  the  program  attributes  in  Table  2  may  make  a  program  more 
vulnerable  to  immature  development  processes.  These  six  attributes  are  a  critical  subset  of  risk- 
inducing  program  characteristics.  Other  attributes  of  the  program  may  also  indicate  vulnerabilities 
to  immature  supplier  development  processes. 
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Table  2:  Factors  Driving  the  Need  for  Process  Capability 


Program  Attribute 

Discussion 

Effect  of  Immature  Development 

Processes 

Complexity 

This  attribute  includes  factors  such 
as  the  number  and  complexity  of 
interfaces,  integration  complexity,  the 
number  and  diversity  of  stakeholders, 
software  reuse,  the  degree  of 
interoperability  with  other  systems, 
and  domain  complexity. 

Immature  development  processes  reduce  the 
program’s  ability  to  deal  with  the  planning, 
execution,  and  oversight  of  complex 
programs. 

Size 

Program  size  indicators  include  the 
number  of  developmental 
components,  the  number  of  acquirer 
and  supplier  teams  participating  in 
the  development  lifecycle,  and  the 
number  of  user,  system,  or  software 
requirements. 

Large  programs  require  geometrically 
increasing  lines  of  communication  for 
oversight  and  stakeholder  involvement  in 
various  lifecycle  development  activities. 
Immature  processes  typically  result  in 
inaccurate  or  late  communication,  poor 
planning,  a  lack  of  teamwork,  and  rework. 

Lack  of  precedent 

This  attribute  includes  five  aspects  of 

the  program: 

1 .  Has  this  system  functionality 
been  developed  before  with 
these  quality  attributes? 

2.  How  mature  is  the  technology? 

3.  Has  the  target  technology  been 
applied  in  this  domain  before? 

4.  Does  this  team  have  experience 
with  this  type  of  system? 

5.  Are  disparate  systems  or 
technologies  brought  together  to 
provide  a  new  integrated 
capability? 

Immature  processes  can  fail  under  the 
pressure  of  responding  to  unprecedented 
challenges.  Mature  programs  with  highly 
disciplined  team  members  have  a  better 
chance  of  rising  to  the  challenge. 

Dependence  on  software 

This  attribute  can  generally  be 
described  by  its  components: 
proportion  and  reliance.  Proportion 
reflects  the  relative  investment  in 
software  development  versus 
hardware  or  systems  development. 
Reliance  indicates  functional 
dependence  of  the  system  on  the 
software. 

Development  of  a  system  that  is  software 
dependent  imposes  challenges  at  the  system, 
software,  and,  potentially,  hardware  levels. 
Immature  processes  may  result  in  a 
breakdown  of  basic  engineering  functions, 
such  as  requirements  allocation,  the 
development  of  coherent  architectures,  and 
system  integration  and  testing. 

Schedule  or  funding 
constraints 

This  attribute  reflects  program 
imperatives  to  deliver  a  useful 
capability  to  the  customer  for  a 
limited  expenditure.  "Hard”  delivery 
dates  and  inadequate  funds  stress 
both  the  acquirer  and  the  supplier 
team. 

Programs  that  exhibit  immature  processes 
have  little  or  no  capability  to  react  reasonably 
to  schedule  or  funding  stress,  to  properly 
manage  expectations,  or  to  effectively 
manage  the  resulting  risks. 

Requirements  instability 

There  is  always  some  level  of 
requirements  evolution  for  complex, 
unprecedented  systems.  However,  if 
user  needs  or  system  requirements 
are  highly  volatile,  either  through  lack 
of  definition  or  in  response  to 
changing  customer  needs,  excessive 
requirements  variation  will  occur  at 
the  system  and  software  levels. 

Programs  with  immature  processes  often  find 
it  difficult  to  respond  to  a  changing 
requirements  base  from  either  an  engineering 
or  management  perspective. 
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Table  3  is  a  diagnostic  tool  that  the  acquirer  can  use  to  judge  the  program’s  rating  with  respect  to 
six  attributes.  For  each  of  the  attributes  shown  in  the  left  column,  the  program  is  rated  low, 
medium,  or  high.  For  example,  if  schedule  urgency  is  driving  the  program  to  be  kicked  off  before 
user  needs  are  clearly  understood,  both  Schedule  and  Funding  Constraints  and  Requirements 
Instability  might  be  rated  high. 

Table  3:  Program  Characterization 


Program  Attribute 

Ratings 

Low 

Medium 

High 

Complexity 

Size 

Lack  of  Precedent 

Dependence  on  Software 

Schedule  or  Funding  Constraints 

Requirements  Instability 

TOTAL 

The  results  of  this  diagnostic  may  be  interpreted  as  follows: 

•  If  the  program  has  no  high  ratings  and  no  more  than  two  medium  ratings,  it  probably  has  a 
low  dependence  on  process  capability. 

•  If  the  program  has  one  high  rating  or  three  or  more  medium  ratings,  it  probably  has  a 
moderate  dependence  on  process  capability. 

•  If  the  program  has  two  or  more  high  risk  ratings,  it  probably  has  a  high  dependence  on 
process  capability. 

If  the  program  is  rated  as  moderate  or  high,  the  process  capabilities  of  the  suppliers  as  they  are 
applied  during  the  development  effort  can  be  a  critical  factor  in  the  success  of  the  program,  and 
the  approaches  in  this  guidebook  may  be  used  to  alleviate  some  of  the  risk. 

2.2  Understand  the  Relevant  Process  Areas 

In  nearly  every  program,  all  17  of  the  CMMI-DEV  process  areas  related  to  Project  Management, 
Engineering,  and  Support  are  used  at  some  time  during  the  program’s  lifecycle.  Of  the  process 
areas  included  at  maturity  level  2  and  3  related  directly  to  the  execution  of  the  program,  some  are 
more  important  than  others  at  different  phases  in  the  program  lifecycle.  The  capability  of  these 
processes  at  the  time  when  they  are  most  needed  determines  the  risk  that  they  pose  to  the 
program.  While  all  process  areas  must  be  planned  for  early  in  the  lifecycle  and  can  apply 
throughout  the  lifecycle,  it  may  be  beneficial  for  some  to  begin  earlier  in  the  program  than  others, 
as  illustrated  in  Figure  3: 
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Figure  3:  Notional  Time  Sequencing  of  Process  Areas  Mapped  to  Development  Lifecycle 
The  process  areas  of  interest  and  their  corresponding  purpose  statements  are  as  follows: 


•  Configuration  Management:  to  establish  and  maintain  the  integrity  of  work  products  using 
configuration  identification,  configuration  control,  configuration  status  accounting,  and 
configuration  audits 

•  Decision  Analysis  and  Resolution:  to  analyze  possible  decisions  using  a  formal  evaluation 
process  that  evaluates  identified  alternatives  against  established  criteria 

•  Integrated  Project  Management:  to  establish  and  manage  the  project  and  the  involvement  of 
the  relevant  stakeholders  according  to  an  integrated  and  defined  process  that  is  tailored  from 
the  organization’s  set  of  standard  processes 

•  Measurement  and  Analysis:  to  develop  and  sustain  a  measurement  capability  that  is  used  to 
support  management  information  needs 

•  Process  and  Product  Quality  Assurance:  to  provide  staff  and  management  with  objective 
insight  into  processes  and  associated  work  products 

•  Product  Integration:  to  assemble  the  product  from  the  product  components,  ensure  that  the 
product,  as  integrated,  functions  properly,  and  deliver  the  product 

•  Project  Monitoring  and  Control:  to  provide  an  understanding  of  the  project’s  progress  so  that 
appropriate  corrective  actions  can  be  taken  when  the  project’s  performance  deviates 
significantly  from  the  plan 

•  Project  Planning:  to  establish  and  maintain  plans  that  define  projecf  activities 

•  Requirements  Development:  to  produce  and  analyze  customer,  product,  and  product 
component  requirements 
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•  Requirements  Management:  to  manage  the  requirements  of  the  project’s  products  and  product 
components  and  to  identify  inconsistencies  between  those  requirements  and  the  project’s 
plans  and  work  products 

•  Risk  Management:  to  identify  potential  problems  before  they  occur  so  that  risk-handling 
activities  can  be  planned  and  invoked  as  needed  across  the  life  of  the  product  or  project  to 
mitigate  adverse  impacts  on  achieving  objectives 

•  Supplier  Agreement  Management:  to  manage  the  acquisition  of  products  from  suppliers 

•  Technical  Solution:  to  design,  develop,  and  implement  solutions  to  requirements.  Solutions, 
designs,  and  implementations  encompass  products,  product  components,  and  product-related 
lifecycle  processes  either  singly  or  in  combination  as  appropriate. 

•  Validation:  to  demonstrate  that  a  product  or  product  component  fulfills  its  intended  use  when 
placed  in  its  intended  environment 

•  Verification:  to  ensure  that  selected  work  products  meet  their  specified  requirements 


These  process  areas  align  well  with  the  technical  and  technical  management  processes  described 
in  Section  4.2  of  the  Defense  Acquisition  Guidebook  (DAG),  Chapter  4,  “Systems  Engineering” 
[DoD  2011a].  (See  Appendix  E  for  more  information  about  mapping  between  DAG  Chapter  4 
processes  and  CMMl-DEV  process  areas.)  According  to  Section  4.5.1  of  the  DAG,  the  systems 
engineering  plan  (SEP)  will  include  the  following: 

•  the  overall  requirements  for  the  program 

•  the  systems  engineering  processes  to  be  applied  in  the  program 

•  the  integration  of  systems  engineering  into  the  program’s  integrated  product  teams  (IPTs) 

•  the  system’s  technical  baseline  approach 

•  event-driven  timing,  conduct,  success  criteria,  and  expected  products  of  technical  reviews 

•  the  synchronization  with  related  systems 

•  human  systems  integration  planning 

•  participants  in  the  systems  engineering  process 

•  systems  engineering  processes  and  products 

•  facilities  enabling  systems  engineering 

•  systems  engineering  event  timing 

•  systems  engineering  decision  rationale 

•  tools  enabling  systems  engineering 

The  systems  engineering  processes  to  be  applied  in  the  program  (e.g.,  those  drawn  from  a 
standard,  a  capability  maturity  model,  or  the  contractor’s  process)  include  a  description  of  how 
the  processes  will  be  implemented  and  tailored  to  meet  individual  acquisition  phase  objectives. 
The  SEP  also  includes  an  explanation  of  how  the  systems  engineering  processes  will  support  the 
technical  and  programmatic  products  required  of  each  phase.  Section  4.2  {Systems  Engineering 
Processes:  How  Systems  Engineering  is  Conducted)  and  section  4.3  {Systems  Engineering 
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Activities  in  the  System  Life  Cycle)  provide  a  roadmap  of  how  systems  engineering  processes  can 
be  applied  to  an  acquisition  program  throughout  the  system  life  cycle. 

2.3  Assess  and  Prioritize  the  Criticai  Process  Areas  for  the  Program 

Based  on  the  previous  discussion  and  the  context  of  the  envisioned  program,  the  acquirer  can 
determine  which  process  areas  are  critical.  Two  methods  are  described  in  this  section  that  can  be 
used  to  assist  the  acquirer  with  this  determination.  The  objective  is  to  assess  and  prioritize  the 
critical  process  areas  of  the  program,  using  CMMI-DEV  as  the  reference  model.  Once  identified, 
the  program’s  critical  process  areas  can  be  addressed  in  the  RFP. 

Supplier  processes  that  relate  to  those  critical  areas  can  then  be  evaluated  in  relationship  to  the 
risk  that  poorly  defined  or  executed  processes  can  create. 

2.3.1  Self  Assessment 

A  self-assessment  of  the  program’s  critical  process  areas  can  be  performed  as  part  of  acquisition 
strategy  development.  Knowing  the  critical  process  areas  helps  to  identify  the  process  capabilities 
that  a  successful  supplier  must  possess.  This  method  requires  that  the  acquirer  has  basic  process 
knowledge.  This  knowledge  should  be  resident  within  the  acquirer’s  office  under  normal 
circumstances.  (See  the  “Self-Assessment  Questionnaire”  on  page  34  in  Appendix  C  for 
implementation  details.) 

2.3.2  Facilitated  Assessment 

A  facilitated  assessment  of  the  program’s  critical  process  areas  can  be  used  as  an  alternative  to  the 
self-assessment  method.  A  facilitated  assessment  may  be  performed  by  a  trained  and  qualified 
outside  agency  with  both  domain  and  process  expertise.  This  method  requires  sufficient  funding 
and  time  for  external  support.  (See  “What  to  Do”  in  the  “Self-Assessment  Questionnaire”  on  page 
37  in  Appendix  B  for  implementation  details.) 
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3  Leverage  the  Process  Capabilities  of  the  Supplier 


Chapter  2  described  how  to  identify  process  areas  that  are  critical  to  a  program  and  how  to 
detennine  what  to  concentrate  on  when  considering  the  supplier’s  approaches.  This  chapter 
includes  methods  for  gathering  information  about  the  capabilities  of  suppliers  to  perform 
development  activities.  The  acquirer  can  use  these  methods  to  determine  organizational  strengths, 
weaknesses,  or  risks  associated  with  a  supplier’s  approach  and  organizational  processes.  These 
methods  are  fully  independent  and  may  be  used  individually  or  in  combination.  Each  method  is 
useful  in  limiting  process  performance  risk  within  an  acquisition,  but  the  acquirer  can  balance  the 
need  for  information  with  the  resources  available  to  review  that  information.  Table  4  at  the  end  of 
this  chapter  provides  guidance  on  selecting  the  methods  that  may  be  most  appropriate  for  the 
acquisition  program. 

Once  the  acquirer  selects  one  or  more  of  these  methods,  they  have  to  be  enabled  in  an  RFP,  the 
contract,  or  in  some  other  manner.  Examples  of  sample  evaluation  criteria  and  instructions  that 
apply  to  Sections  M  and  L  of  the  RFP  are  available  in  the  Guide  for  Integrating  Systems 
Engineering  into  DoD  Acquisition  Contracts  [DoD  2006d].  The  sample  language  in  that  guide 
should  be  adapted  to  align  with  specific  selections  of  the  methods  described  below. 

3.1  Consider  Process  Proposals  in  Critical  Areas 

One  way  for  the  acquirer  to  understand  the  supplier’s  plans  for  process  application  to  the  program 
is  to  request  and  evaluate  process  proposals  in  areas  critical  to  the  program.  When  preparing  a 
proposal,  the  offeror^  plans  and  estimates  the  work  that  will  be  done  on  the  program.  This 
estimate  is  based  on  the  offeror’s  intended  methods  for  executing  the  work  (i.e.,  its  work 
processes).  The  acquirer  can  ask  for  a  detailed  description  of  processes  in  areas  critical  to  program 
success,  as  identified  in  Chapter  2.  Each  offeror’s  response  to  this  request  illustrates  how  its 
processes  are  adapted  to  and  applied  to  the  program. 

Offerors  with  a  strong  process  focus  and  capable  processes  (i.e.,  maturity  level  3  or  capability 
level  3  as  defined  by  CMMI-DEV)  already  have  process  descriptions.  Offerors  who  have  not 
implemented  CMMI-DEV  can  still  participate  by  submitting  the  specific  processes  that  they  plan 
to  use  on  the  program. 

The  acquirer  must  balance  the  desire  for  detailed  information  against  the  impact  that  a  large 
volume  of  information  will  have  on  the  evaluation  effort.  Concentrating  on  only  the  most  critical 
process  areas,  as  defined  in  Chapter  2,  may  help  achieve  such  a  balance. 

The  information  supplied  by  the  offeror  in  process  proposals  that  address  the  most  critical  process 
areas  can  be  formally  evaluated  as  part  of  the  source  selection.  The  SCAMPI  Class  C  process 
appraisal  method  can  be  used  to  review  the  process  information  from  the  proposals  such  as 
process  definitions  and  descriptions,  templates,  and  sample  work  products.  The  appraisal  team 


An  offeror  is  an  organization  competing  for  a  contract.  An  offeror  becomes  a  supplier  upon  being  awarded  a 
contract  to  provide  a  product  or  service. 
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uses  this  process  information  and  the  practices  in  CMMI-DEV  as  a  reference  to  identify  strengths 
and  weaknesses  in  the  offeror’s  proposed  processes. 

Furthermore,  certain  aspects  of  the  proposed  processes  can  then  be  captured  in  the  contract, 
creating  an  obligation  for  the  chosen  supplier  to  implement  them  and  providing  a  basis  for 
contract  monitoring.  (See  “Consider  Process  Proposals  in  Critical  Areas”  on  page  40  in  Appendix 
C  for  implementation  details.) 

3.2  Consider  Integrated  Master  Plan  Documentation  of  Proposed  Processes 

An  integrated  master  plan  (IMP)  that  describes  how  the  offeror  intends  to  execute  the  program  is 
linked  to  the  offeror’s  integrated  master  schedule  (IMS)  and  uses  the  same  program  work 
breakdown  structure  (WBS)  used  to  prepare  the  basis  of  estimate  (BoE)  documentation.  If  the 
proposed  processes  are  included  in  the  IMP,  the  acquirer  will  have  the  means  to  verify  process 
execution  without  additional  process  monitoring  actions. 

The  work  described  in  this  set  of  documentation  represents  how  each  offeror  has  planned  to 
execute  the  program  and,  as  such,  represents  a  commitment  to  the  incorporation  of  its  work 
processes  into  program  execution.  The  value  of  the  complete  package  is  that  it  enables  the 
acquirer  to  review  each  offeror’s  approach  to  implementing  the  program  in  a  holistic  sense, 
especially  if  the  offeror  is  asked  to  map  its  proposed  activities  to  its  processes  and  to  the  critical 
process  areas  identified  by  the  program.  This  mapping  can  be  done  with  little  additional 
documentation  and  allows  the  evaluator  to  see  how  well  the  offeror’s  processes  will  incorporate 
into  the  program  plan.  More  importantly,  the  acquirer  can  use  it  to  evaluate  the  degree  to  which 
each  offeror  intends  to  invoke  its  processes  and  practices.  This  is  especially  important  if  the 
offeror  is  asked  to  provide  that  information  in  the  IMP  as  well  as  provide  a  map  of  its  process  set 
to  the  program’s  identified  critical  process  areas  in  the  IMP  or  elsewhere  in  the  proposal.  This 
evaluation  provides  a  valuable  assessment  of  the  degree  to  which  each  offeror  can  be  expected  to 
execute  its  processes  after  contract  award  and  allows  the  evaluator  to  assign  an  appropriate  risk  to 
each  offeror.  (See  “Consider  Integrated  Master  Plan  Documentation  of  Proposed  Processes”  on 
page  42  in  Appendix  C  for  implementation  details.) 

General  guidance  on  the  IMP  and  IMS  can  be  found  in  the  Integrated  Master  Plan  and  Integrated 
Master  Schedule  Preparation  and  Use  Guide  [DoD  2005]. 

3.3  Consider  Incorporation  of  Process  Reviews  Into  Proposed  Program 
Schedules 

Organizations  that  have  institutionalized  their  processes  should  periodically  review  or  audit  them. 
Incorporation  of  these  process  reviews  into  the  offeror’s  IMS  and  IMP  provides  visibility  into  the 
rollout  of  the  offeror’s  processes  and  work  products  for  critical  program  processes. 

The  acquirer  can  request  that  the  offeror  include  its  planned  process  reviews  in  its  IMP  and  IMS, 
including  a  list  of  process  work  products  for  the  program’s  critical  process  areas,  which  will  be 
included  in  the  earned  value  management  system  (EVMS)  and  initial  baseline  review  (IBR).  (See 
“Consider  Incorporation  of  Process  Reviews  into  Proposed  Program  Schedules”  on  page  43  in 
Appendix  C  for  implementation  details.) 
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3.4  Consider  the  Approach  to  Integration  with  Subcontractor  Processes 


Process  relationships  between  the  prime  contractor  (supplier)  and  major  development 
subcontractors  can  be  a  potential  source  of  risk  to  the  program.  Therefore,  it  is  important  for  the 
acquirer  to  know  and  understand  the  risks  arising  from  the  integration  of  prime  contractor  and 
subcontractor  processes  to  properly  monitor  and  manage  these  risks.  The  RFP  can  request  a  plan 
from  the  prime  contractor  addressing  this  subject,  and  it  can  ensure  that  all  activities  and  risks  are 
captured  in  the  IMP,  IMS,  and  risk  management  plan  (RMP).  (See  “Consider  the  Approach  to 
Integration  with  Subcontractor  Processes”  on  page  44  in  Appendix  C  for  implementation  details.) 

3.5  Consider  the  Approach  to  Integration  with  Acquirer  Processes 

Relationships  between  the  acquirer  and  supplier  processes  can  be  a  source  of  risk  to  the  program. 
Since  the  acquirer  and  the  supplier  have  significantly  different  roles  and  responsibilities  in  an 
acquisition  program,  their  processes  differ  significantly.  However,  in  areas  where  responsibility 
for  processes  is  shared  or  where  significant  interaction  exists  between  the  acquirer  and  supplier 
processes,  it  is  desirable  to  ensure  that  supplier  processes  critical  to  the  program  (identified  in 
Chapter  2)  are  compatible  with  the  acquirer  processes.  Process  compatibility  requires  the 
following: 

•  a  common  vocabulary  (e.g.,  what  is  the  meaning  of  a  high-risk  exposure) 

•  consistent  and  compatible  process  objectives 

•  efficient  and  clear  bidirectional  communication  of  process  performance 

•  a  clear  understanding  of  the  acquirer  and  supplier  processes 

•  alignment  of  supplier  lifecycle  phase  deliverables  and  work  products  with  the  acquirer’s 
lifecycle  expectations 

Examples  of  process  integration  include  the  following: 

•  The  acquirer  and  the  supplier  share  responsibility  for  risk  management.  To  ensure  effective 

risk  management  for  the  program,  common  definitions  can  be  established  for  risk  impact,  risk 

probability,  etc.  Furthermore,  guidelines  can  be  established  to  help  determine  which  risks  are 

the  responsibility  of  the  supplier,  which  are  the  responsibility  of  the  acquirer,  and  which  are 

4 

shared.  Issues  such  as  the  frequency  of  risk  reporting  must  also  be  addressed. 

•  Both  the  acquirer  and  the  supplier  will  perform  requirements  management  and  requirements 
development  activities.  The  acquirer  will  translate  stakeholder  needs  (e.g.,  user  requirements) 
into  contract  requirements.  These  requirements  must  be  accurate,  complete,  and  consistent; 
they  must  also  be  baselined  and  managed.  The  supplier  takes  the  contract  requirements  and 
derives  product-based  requirements  from  them.  These  requirements  must  also  be  baselined 
and  managed.  To  ensure  satisfaction  of  user  needs,  traceability  must  be  maintained  from  the 
user  needs  through  the  product-based  requirements. 


For  further  information  on  the  alignment  of  acquirer  and  supplier  risk  management  activities,  see  the  Risk  Man¬ 
agement  Guide  for  DoD  Acquisition  [DoD  2006c]. 
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•  Configuration  management  (CM)  is  a  shared  responsibility  between  the  acquirer  and  the 
supplier;  both  must  manage  the  configuration  of  program  documents  to  ensure  accurate 
communication  and  real-time  understanding  of  program  status.  Coordination  of  CM  processes 
creates  a  consistent  baseline  on  which  both  the  acquirer  and  the  supplier  can  base  their  efforts. 

The  acquirer  can  monitor  and  manage  any  risks  arising  from  the  integration  of  its  processes  and 
the  supplier  processes  once  they  are  known.  (See  “Consider  Approach  to  Integration  with 
Acquirer  Processes”  on  page  45  in  Appendix  C  for  implementation  details.) 

3.6  Consider  Recent  Appraisal  Results 

Many  organizations  perform  formal  process  appraisals  to  understand  their  process  capabilities  and 
degree  of  compliance  with  CMMI.  In  these  appraisals,  a  representative  sampling  of  programs 
within  the  organization  is  examined  to  identify  evidence  that  best  practices  as  defined  in  CMMI- 
DEV  were  deployed. 

Results  of  previous  organizational  appraisals  (within  the  past  three  years)  can  be  requested  and 
collected  from  each  prime  contractor  and  each  subcontractor  that  will  perform  major  development 
activities.  Knowledge  and  understanding  of  appraisal  results  can  help  the  acquirer  identify  the 
process  capabilities  of  the  supplier  in  relationship  to  the  critical  process  areas  identified  in 
Chapter  2. 

The  acquirer  should  look  for  answers  to  the  following  questions  from  data  supplied  on  the 
Appraisal  Disclosure  Statement  (ADS): 

•  Is  the  organization  named  in  the  ADS  the  same  organization  that  is  bidding  on  and  will 
execute  the  program?  {Given  that  regular  consolidations  and  reorganizations  occur,  if  the 
organization  is  not  the  same,  the  supplier  should  have  provided  a  clear  explanation  of  how 
the  appraised  organization  relates  to  the  bidding  organization.  The  acquirer  will  have  to  be 
careful  to  validate  that  the  proposed  development  sites  involved  also  relate  to  the  appraised 
site  regardless  of  the  organizational  name.) 

•  How  many  programs  were  included  in  the  appraisal  and  were  they  relevant  to  your  current 
acquisition? 

•  Was  the  appraisal  performed  less  than  three  years  ago? 

•  Which  process  areas  were  included  in  the  appraisal? 

Additional  questions  that  can  be  asked  about  the  appraisal  findings  include  the  following: 

•  Do  the  appraisal  findings  declare  Supplier  Agreement  Management  (SAM)  process  area  not 
applicable  (NA)?  (Declaring  SAM  as  not  applicable  may  indicate  that  the  appraised 
organization  does  not  have  a  practice  for  managing  subcontractors.  If  the  supplier  intends  to 
manage  subcontractors  in  the  project  being  acquired,  then  there  may  be  an  acquisition  risk.) 

•  Will  weaknesses  noted  in  the  appraisal  findings  have  a  significant  impact  on  the  program? 

(See  “Consider  Recent  Appraisal  Results”  on  page  46  in  Appendix  C  for  implementation  details.) 
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3.7  Consider  Historical  Data  on  Process  Performance 


Organizations  that  have  institutionalized  their  processes  should  collect  and  maintain  data  on  the 
historical  use  of  their  processes  on  programs.  Data  may  include  statistics  on  the  following: 

•  breadth  of  process  application  across  the  organization  (e.g.,  what  percentage  of  programs 
employ  standard  or  tailored  versions  of  the  organization’s  standard  processes) 

•  consistency  of  process  application  (e.g.,  what  percentage  of  processes  applied  to  programs  are 
modified  from  the  organization’s  standard  processes) 

•  program  compliance  with  the  defined  processes  (e.g.,  metrics  resulting  from  process 
monitoring  or  process  audits) 

The  acquirer  may  request  and  evaluate  any  relevant  data  on  historical  process  performance.  As 
part  of  the  evaluation  of  the  past  performance  of  source  selection,  the  acquirer  may  ask  the  offeror 
to  identify  the  historical  performance  of  selected  processes  for  the  number  of  completed  programs 
the  offeror  submitted  for  consideration.  In  addition,  the  acquirer  may  request  that  past 
performance  data  be  submitted  for  ongoing  programs  coincident  with  the  present  bid,  recognizing 
that  some  of  these  programs  may  not  be  sufficiently  advanced  in  their  lifecycle  to  have 
completion  data  available.  For  a  DoD  acquisition,  the  acquirer  may  also  contact  DCMA  to  obtain 
past  performance  assessment  information  on  suppliers.^  (See  “Consider  Historical  Data  on 
Process  Performance”  on  page  50  in  Appendix  C  for  implementation  details.) 

3.8  Consider  Recent  Post-Award  Appraisal  Data  from  Other  Programs 

Over  the  past  few  years,  several  government  acquisition  programs  planned  for  and  executed  post¬ 
award  appraisals  within  the  winning  supplier  or  supplier  team.  These  appraisals  were  performed 
by  trained  and  authorized  appraisal  teams  consisting  of  members  from  the  acquisition 
organization,  the  supplier  organizations,  and  independent  members.  The  purpose  of  these 
appraisals  was  to  address  the  process-related  risks  of  the  appraised  program.  These  appraisals  are 
limited  to  the  scope  of  the  program  and  the  portion  of  the  organization  executing  the  program. 

The  purpose  is  not  to  evaluate  an  organization’s  process  capability,  but  to  identify  risks 
introduced  into  the  program  by  process  performance. 

These  appraisals  have  been  performed  at  several  stages  of  the  program  lifecycle  and,  therefore, 
may  or  may  not  be  useful  to  the  program: 

•  Some  were  performed  near  the  end  of  one  phase  of  system  development,  preceding  a  down- 
select®  to  provide  input  to  the  source  selection  process. 

•  Some  were  performed  shortly  after  contract  award  to  encourage  the  supplier  to  instantiate  the 
necessary  processes  on  the  program  quickly. 

•  Some  were  performed  later  in  the  development  lifecycle  to  monitor  the  supplier’s 
conformance  to  process  performance  commitments. 


See  also  the  Guide  for  Integrating  Systems  Engineering  into  DoD  Acquisition  Contracts  [DoD  2006d]. 

Downselect  is  an  acquisition  term  that  describes  the  process  of  determining  a  winning  bid  after  two  or  more 
competing  contractors  demonstrate  their  designs  or  prototypes.  Often,  to  alleviate  program  risk,  an  acquisition 
strategy  will  award  two  contracts  to  two  offeror  teams  with  competing  designs.  Downselect  is  the  term  used 
when  one  design  or  prototype  is  finally  selected  after  it  is  proven  to  be  superior  to  the  others. 
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(See  “Consider  Recent  Post-Award  Appraisal  Data  from  Other  Programs”  on  page  52  in 
Appendix  C  for  implementation  details.) 

3.9  Consider  Process  Compliance  Evaluations  During  Execution 

The  degree  to  which  a  program  complies  with  the  organization’s  set  of  standard  processes  is 
extremely  important  and  indicates  how  the  offeror  might  perform  on  the  program.  The 
compliance  of  the  offeror’s  standard  processes  should  be  measured  early  in  each  program’s 
lifecycle  then  measured  again  at  reasonable  intervals  throughout  the  lifecycle. 

During  the  execution  of  the  program,  the  acquirer  can  monitor  the  implemented  processes  to 
ensure  the  program  is  benefiting  from  the  process  capabilities  of  the  supplier.  There  are  several 
ways  to  monitor  a  supplier’s  process  performance: 

•  If  organizations  include  process  compliance  evaluations  as  part  of  regular  quality  assurance 
reviews,  the  acquirer  should  obtain  and  review  the  collected  data. 

•  If  organizations  evaluate  process  performance  as  part  of  a  global  mission  assurance  review 
process,  the  acquirer  should  obtain  and  review  the  collected  data. 

•  For  DoD  acquisitions,  the  acquirer  should  contact  the  local  DCMA  office  to  request 
performance  of  independent  process  reviews. 

(See  “Consider  Process  Compliance  Evaluations  during  Execution”  on  page  52  in  Appendix  C  for 
implementation  details.) 

3.10  Consider  the  Reporting  of  Process  Statistics 

Receiving  reports  of  process  statistics  can  provide  the  acquirer  with  near-continuous  insight  into 
the  supplier’s  process  capability.  Organizations  that  have  institutionalized  their  processes 
normally  monitor  and  control  those  processes.  As  a  fundamental  part  of  the  supplier’s  own 
process  monitoring  activities,  it  collects,  analyzes,  and  uses  process  metrics  to  ensure  its  processes 
are  being  applied,  are  appropriate,  and  are  being  used  as  a  basis  for  improvement.  As  part  of  the 
metrics  program,  the  acquirer  can  include  reporting  of  process  performance  metrics  (e.g.,  process 
compliance  data  or  defect  rate  data)  in  the  RFP.  Ensure  that  this  reporting  requirement  is 
incorporated  into  the  contract.  (See  “Consider  the  Reporting  of  Process  Statistics”  on  page  53  in 
Appendix  C  for  implementation  details.) 

3.1 1  Consider  Access  to  Selected  Process  Artifacts 

Executing  a  process  results  in  work  products,  also  called  process  artifacts.  Examining  these  work 
products  can  provide  insight  into  process  performance.  For  example,  if  a  peer  review  process  calls 
for  the  publication  of  meeting  minutes,  the  presence  (or  absence)  of  those  minutes  can  give  some 
idea  of  the  number  of  peer  reviews,  the  subjects  of  peer  reviews,  and  even  the  quality  of  the  peer 
reviews.  Likewise,  the  acquirer  can  evaluate  the  risk  management  process  by  periodically 
examining  the  risk  database  to  reveal  information  such  as  the  following: 

•  Currency  of  the  program  risks:  Are  risks  being  added  and  resolved  on  an  ongoing  basis? 

•  Breadth  of  the  risk  management  process  deployment:  Who  is  identifying  and  managing  risks? 
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Many  process  artifacts  mentioned  in  CMMI-DEV  are  plans  or  other  documents  that  are  typically 
part  of  any  development  program.  The  acquirer  can  decide  which  of  the  artifacts  should  be 
delivered  formally  and  which  should  only  require  informal  access.  (See  “Consider  Access  to 
Selected  Process  Artifacts”  on  page  54  in  Appendix  C  for  implementation  details.) 

3.12  Consider  Conducting  Pre-Award  Process  Appraisais 

While  the  results  of  previous  appraisals  using  CMMI-DEV  can  be  helpful  in  identifying  the 
supplier’s  process  capabilities,  they  may  not  be  sufficient  for  a  pending  program’s  needs  because 
of  the  following  reasons: 

•  The  portion  of  the  organization  bidding  on  this  program  was  not  involved  in  the  previous 
appraisal. 

•  The  programs  evaluated  in  the  previous  appraisal  were  not  in  the  same  technology  domain  as 
this  program. 

•  Processes  critical  to  this  program  were  not  appraised  in  the  previous  appraisal. 

If  sufficient  information  on  process  capability  has  not  been  submitted  by  the  offeror  to  make  the 
acquirer  comfortable  in  the  source  selection,  it  may  be  helpful  to  conduct  an  appraisal  of  the 
offeror’s  organizations  and  locations  performing  the  work  on  the  program.  Such  an  appraisal  can 
be  conducted  prior  to  the  release  of  the  RFP  in  an  advisory  multi-step  process^  or  after  the  REP 
release  as  part  of  the  evaluation  of  the  source  selection.  While  it  is  generally  not  practical  to 
conduct  a  comprehensive  appraisal  of  all  process  areas  within  the  organization,  it  is  feasible  to 
conduct  an  appraisal  of  a  few  process  areas  as  a  means  of  filling  gaps  in  understanding  the 
offeror’s  process  capabilities.  A  SCAMPI  Class  B  or  Class  C  appraisal  may  be  used. 

An  appraisal  conducted  prior  to  or  during  source  selection  will  not  evaluate  the  specific  team  and 
the  specific  tailored  processes  that  will  execute  the  program.  Although  CMMI-DEV  requires  the 
early  deployment  of  processes  on  new  programs,  in  most  cases  the  program  team  will  not  exist 
and  the  processes  will  not  be  implemented  until  after  contract  award.  These  process  appraisals 
only  allow  the  acquirer  to  evaluate  the  process  capabilities  of  teams  similar  to  the  envisioned 
program  team,  working  on  a  similar  program  in  the  same  organization  and  environment  that  the 
actual  program  will  have.  (See  “Consider  Conducting  Pre-Award  Process  Appraisals”  on  page  55 
in  Appendix  C  for  implementation  details.) 

3.13  Consider  Conducting  Post-Award  Process  Appraisais 

If  the  program’s  dependence  on  process  capability  was  determined  to  be  significant  using  the 
methods  prescribed  in  Chapter  2,  the  acquirer  can  require  that  the  winning  supplier  undergo  one 
or  more  process  appraisals  of  the  program’s  critical  process  areas.  Typically,  these  appraisals  are 
performed  using  a  SCAMPI  Class  B  method  that  only  addresses  the  processes  proposed  for  the 
program. 

The  acquirer  can  require  the  supplier  to  incorporate  process  appraisals  directed  by  the  acquirer 
into  its  proposed  IMS  and  IMP  and  to  plan  for  cooperation  with  these  appraisals.  The  acquirer 
also  can  provide  instructions  on  approximate  timeframes  for  the  conduct  of  each  appraisal  as  well 


This  process  is  expiained  on  Defense  Acquisition  University’s  website  [DoD  2006a]. 
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as  information  on  how  far  in  advance  the  supplier  will  be  notified.  Typically,  two  process 
appraisals  are  performed — one  shortly  after  contract  award  and  a  second  later  in  the  program, 
possibly  before  a  major  program  decision  point.  The  first  appraisal  establishes  a  benchmark  of 
process  performance  and  can  be  used  to  encourage  the  supplier  to  expedite  process  instantiation 
on  the  program.  The  second  appraisal  conducted  in  advance  of  a  key  event  can  help  the  acquirer 
assess  the  quality  of  the  efforts  and  products  contributing  to  that  event.  (See  “Consider 
Conducting  Post-Award  Process  Appraisals”  on  page  56  in  Appendix  C  for  implementation 
details.) 

3.14  Consider  Process-Based,  Outcome-Oriented  Award  Fee  Determinants 

A  risk  mitigation  technique,  oriented  to  encourage  the  winning  offeror  to  embrace  process 
implementation,  is  to  negotiate  adding  an  award  fee  determination  based  on  process  adoption  and 
improvement.  If  process  performance  is  included  as  part  of  the  contract  award  fee  conditions  in 
the  RFP,  then  the  acquirer  can  set  the  details  of  the  award  fee  conditions  for  each  offeror  as  part 
of  the  contract  negotiations.  The  acquirer  must  keep  any  award  fee  related  to  process  performance 
in  balance  with  program  outcomes.  According  to  existing  guidance  on  contracts  that  contain 
award  fee  conditions,  “it  is  imperative  that  award  fees  be  tied  to  identifiable  interim  outcomes, 
discrete  events  or  milestones,  as  much  as  possible”  [USD  2006]. 

For  DoD  acquisitions,  the  acquirer  may  contact  the  local  DUMA  office  to  request  its  support  with 
the  evaluation  of  award  fee  criteria  implementation  during  contract  execution.  (See  “Consider 
Process-Based,  Outcome-Oriented  Award  Fee  Determinants”  on  page  57  in  Appendix  C  for 
implementation  details.) 

The  methods  presented  in  this  chapter  are  not  mutually  exclusive.  Any  or  all  of  them  may  be  used 
on  a  program.  Table  4  provides  guidance  on  selecting  the  methods  most  appropriate  given 
program  needs. 
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Table  4:  Opportunities  to  Leverage  Supplier  Capabilities — Selection  Guidance 


Title 

Purpose 

Prerequisites 

Outputs 

Impact 

3.1 

Consider  Process 
Proposals  in  Critical 
Areas 

Pages  15  and  40 

Identify  and  understand  the  offeror’s 
intended  application  of  processes  to 
the  program 

Establish  a  basis  for  a  contractual 
obligation  for  process  performance 

Provide  insight  into  artifacts  available 
to  the  acquirer  for  process  monitoring 

Gain  insight  into  organizational 
process  capability 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer  staff  or 
external  support  for  process  proposal 
evaluation 

Critical  areas  have  been  identified 

Proposed  processes 

List  of  available  process  artifacts 

Identification  of  gaps  between  offeror 
capabilities  and  program  needs 

Provides  a  basis  for  contractual 
obligation  for  process  performance 

Requires  significant  effort  from  the 
acquirer  and  may  require  significant 
effort  from  the  offeror 

Use  this  method 

•  on  acquisitions  for  which 
dependence  on  process  capability 
is  significant 

•  for  offerors  whose  process 
capabilities  are  largely  unknown  to 
the  acquirer 

•  to  address  only  the  process  areas 
most  critical  to  the  program 

3.2 

Consider  Integrated 
Master  Plan 
Documentation  of 
Proposed 

Processes 

Pages  16  and  42 

Gain  visibility  of  the  degree  to  which 
offerors  are  planning  to  execute  their 
processes  on  the  program  {i.e., 
contractual  commitment  to  proposed 
processes) 

Defined  offeror  processes  and 
identified  artifacts  that  can  be 
referenced  in  the  IMP  and  IMS 

The  acquirer  has  knowledge  or 
external  support  available  for 
evaluation  of  the  IMS  and  IMP 

An  integrated  IMP  and  IMS  that 
incorporate  process  elements  into 
the  program  plan  and  schedule 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

3.3 

Consider 

Incorporation  of 
Process  Reviews 
into  Proposed 
Program  Schedules 

Pages  16  and  43 

Provides  the  acquirer  with  visibility 
into  the  deployment  of  processes 
critical  to  the  program 

The  acquirer  has  knowledge  of  the 
expected  execution  of  processes 
critical  to  the  program  within  the 
program's  development  lifecycle 

The  acquirer  has  knowledge  or 
external  support  available  for 
evaluation  of  the  IMS,  IMP,  and 
proposed  process-related  work 
products 

An  IMS  and  IMP  that  include  effort 
and  resources  for  periodic  reviews  of 
processes  critical  to  the  program 

List  of  process-related  work  products 
for  the  processes  critical  to  the 
program 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

A  supplier  at  maturity  level  2  or 
higher  is  required  to  conduct  process 
reviews  under  CMMI  requirements. 
While  placing  it  in  the  schedule  may 
increase  the  formality,  costs 
associated  with  this  approach  should 
not  be  significant. 

3.4 

Consider  the 
Approach  to 
Integration  with 
Subcontractor 
Processes 

Pages  17  and  44 

Assure  adequate  integration  and 
coordination  of  prime  contractor  and 
subcontractor  processes 

Subcontract  relationships 

Subcontractor  management  plan 

List  of  subcontractor  integration  risks 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 
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Title 

Purpose 

Prerequisites 

Outputs 

Impact 

3.5 

Consider  the 
Approach  to 
Integration  with 
Acquirer  Processes 

Pages  17  and  45 

Assure  adequate  integration  and 
coordination  of  the  acquirer  and 
supplier  processes 

Establish  expectations  for 
cooperation  and  integration 

Provide  the  basis  for  creating  a 
contractual  obligation  for  cooperation 
and  integration 

Defined  critical  process  interfaces 

The  supplier  processes  aligned  with 
the  acquirer  processes 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

3.6 

Consider  Recent 
Appraisal  Results 

Pages  18  and  46 

Gain  insight  into  the  organizational 
maturity  and  process  capability  of 
offerors 

Appropriate  appraisal  data  available 
(e.g.,  relevant  scope,  organization, 
domain) 

Identification  of  process  assets 
available  to  the  program 

Identification  of  gaps  between  offeror 
capabilities  and  program  needs 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

Use  on  acquisitions  where  a 
sufficient  subset  of  potential  offerors 
can  provide  appraisal  data 

3.7 

Consider  Historical 
Data  on  Process 
Performance 

Pages  19  and  50 

Understand  and  identify  risks 
associated  with  the  application  of  the 
offeror’s  process  capabilities  to  new 
programs 

Sufficient  data  on  the  application  of 
mature  processes  to  new  programs 

Process  knowledge  and  experience 
on  the  acquirer  staff’s  part  or  external 
support  for  process  proposal 
evaluation 

Insight  into  the  offeror’s  ability  to 
instantiate  organizational  processes 
rapidly  on  new  programs 

Risks  associated  with  the  offeror’s 
consistency  in  applying  mature 
processes  to  new  programs 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

Can  provide  an  extremely  useful 
analysis  that  can  differentiate  offerors 
based  on  their  commitment  to  define 
and  execute  capable  processes  on 
new  programs 

Use  if  there  are  doubts  about  the 
intent  of  offerors  to  define  and  apply 
capable  processes  on  the  program 

3.8 

Consider  Recent 
Post-Award 

Appraisal  Data  from 
Other  Programs 

Pages  19  and  52 

Gain  an  improved  understanding  of 
the  supplier’s  process  execution 
capability  on  recent  programs 

Ideally,  the  offeror  had  post-award 
appraisals  performed  on  recent  and 
similar  programs  within  the 
organization  bidding  this  proposal 

Appraisal  data,  including  the 
following 

•  final  findings  presentation 

.  ADS 

•  action  plans 

The  collection  and  analysis  of  this 
data  provides  increased  confidence 
in  the  abilities  and  risks  of  the  offeror 
as  demonstrated  by  performance  in 
actual  programs. 

Requires  minimal  effort  from  the 
acquirer  and  the  offeror 

When  this  data  is  applicable  and 
available,  use  this  method  on 
acquisitions  where  processes  have 
been  deemed  to  be  a  critical  factor. 
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Title 

Purpose 

Prerequisites 

Outputs 

Impact 

3.9 

Consider  Process 
Compliance 
Evaluations  During 
Execution 

Pages  20  and  52 

Facilitate  contract  monitoring 
activities  later  in  the  program  by 
including  provisions  in  the  contract 

None 

Execution  of  the  method  later  in  the 

program  may  require  one  or  more  of 

the  following: 

•  expertise  in  measurement  and 
analysis  to  monitor  supplier  data 

•  process  and  appraisal  knowledge 
and  experience  on  the  part  of  the 
staff 

•  external  support  for  measurement 
and  analysis 

•  external  support  for  process  and 
appraisal  activities 

Contract  language  establishing  the 
acquirer’s  access  to  relevant  supplier 
data  and  rights  to  perform  program 
process  appraisals 

Enables  later  contract  monitoring, 
requires  minimal  effort  from  the 
acquirer  and  can  greatly  facilitate 
later  monitoring  activities 

Later  execution  of  some  of  these 
activities  (e.g.,  program  process 
appraisals,  data  reviews)  will  impose 
demands  on  the  acquirer;  however, 
judgments  on  proceeding  with  the 
activities  can  be  made  at  that  time 
based  on  program  conditions. 

One  drawback  is  the  slight  increase 
in  cost  to  cover  supplier  support  for 
these  efforts. 

3.10 

Consider  the 
Reporting  of 

Process  Statistics 

Pages  20  and  53 

Ensure  process  compliance  during 
program  execution 

Measurement  and  analysis 
knowledge 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

Process  audit  reports 

Process  compliance  statistics 

Can  provide  significant  benefit  at  a 
low  cost  to  the  acquirer  and  the 
supplier 

A  supplier  at  maturity  level  2  or 
higher  is  required  to  collect  process 
statistics,  so  additional  costs  should 
not  be  significant. 

3.11 

Consider  Access  to 
Selected  Process 
Artifacts 

Pages  20  and  54 

Ensure  process  compliance  during 
program  execution  through 
monitoring  of  process  artifacts 

Measurement  and  analysis 
knowledge 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

Process  artifacts 

Indications  of  process  compliance 

Can  provide  significant  benefit  at  a 
low  cost  to  the  acquirer  and  the 
supplier 

3.12 

Consider 

Conducting  Pre- 
Award  Process 
Appraisals 

Pages  21  and  55 

Evaluate  the  offeror’s  organizational 
process  capability,  or  capability  of 
proposed  or  actual  development 
processes 

Differentiate  the  offerors’  process 
capabilities  during  source  selection 

Reinforce  the  commitment  of  the 
supplier  team  to  process  capability 
after  the  contract  award 

Appraisals  must  be  identified  in  the 
source  selection  or  contract  action. 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

Strengths,  weaknesses,  and  risks  of 
the  offeror’s  organizational 
processes,  proposed  processes,  or 
actual  processes 

A  process  capability  evaluation  of  the 
offeror’s  proposed  process  can 
reveal  lapses  in  intent  or  knowledge 
associated  with  mature  development 
processes.  SCAMPI  Class  C 
appraisals  provide  fairly  inexpensive, 
quick  mechanisms  for  an  in-depth 
understanding  of  the  offeror’s  intent 
that  can  support  comparison  and 
differentiation  of  offerors. 
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Title 

Purpose 

Prerequisites 

Outputs 

Impact 

3.13 

Consider 

Conducting  Post- 
Award  Process 
Appraisals 

Pages  21  and  56 

Ensure  rapid  application  of 
organizational  processes  to  the 
acquirer’s  program 

Ensure  process  compliance  during 
program  execution 

Gain  deeper  insight  into  supplier 
progress 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

Post-award  appraisals  are  enabled  in 
the  contract 

List  of  process-related  risks  to  the 
program 

Supplier  process-compliance  data 

Can  provide  significant  benefit,  but 
the  cost  to  the  acquirer  and  the 
supplier  is  not  trivial 

Use  if  the  supplier  has  a  history  of 
process-related  issues  (e.g.,  slow 
application  of  organizational 
processes,  poor  process 
compliance),  or  if  the  acquirer  has 
other  reasons  to  expect  process 
issues  in  execution 

3.14 

Consider  Process- 
Based,  Outcome- 
Oriented  Award  Fee 
Determinants 

Pages  22  and  57 

Encourage  process  compliance  and 
continuous  improvement 

Process  knowledge  and  experience 
on  the  part  of  the  acquirer’s  staff  or 
external  support  for  process  proposal 
evaluation 

Process-based  outcome-oriented 
award  fee  determinants 

Can  provide  significant  benefit  at  a 
low  cost  to  the  acquirer  and  the 
supplier 
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4  Monitor  Supplier  Process  Performance  after  Award 


This  chapter  provides  guidance  on  how  to  ensure  that  the  supplier  is  effectively  using  capable 
processes  on  the  program,  as  promised  during  solicitation  and  award.  Monitoring  supplier 
execution  after  contract  award  is  a  cornerstone  of  the  acquirer’s  role  as  an  acquisition 
organization. 

Several  methods  can  be  used  to  obtain  the  correct  process  insight  throughout  the  duration  of  the 
contract.  Some  methods  are  inherent  in  a  typical  contract  through  the  submission  and  review  of 
contract  deliverables  that  typically  result  from  process  execution.  Others  involve  deliberate 
review  of  process  documentation  and  artifacts  specifically  involved  in  the  program.  In  addition, 
for  DoD  acquisitions,  DCMA  is  a  source  for  performing  on-site  process  monitoring. 

The  methods  used  are  a  function  of  the  program  risks,  the  leverage  provided  within  the  contract 
structure,  and  the  resources  available  for  risk  reduction.  Artifact  reviews  can  be  conducted  on 
formal  contract  deliverables  already  prescribed  for  the  program.  Other  types  of  reviews  involving 
risk  assessments  or  audits  have  been  proven  to  provide  greater  insight  into  process  performance 
and  execution  risks,  but  often  require  enabling  clauses  in  the  contract  to  execute  them. 

Early  invocation  of  these  methods  can  be  a  means  of  jump-starting  a  program’s  process  execution. 
The  methods  are  independent  and  presented  in  no  particular  order — any  or  all  of  them  may  be 
used. 

4.1  Process  Monitoring  Through  Artifact  Reviews 

Acquirers  regularly  review  deliverables  (e.g.,  contract  data  requirements  lists  [CDRLs],  status 
reports,  audit  reports)  to  evaluate  performance.  These  same  reviews,  when  linked  to  the  supplier’s 
contractual  processes,  can  also  provide  a  validation  of  both  a  supplier’s  adherence  to  its  processes 
and  its  timely  and  effective  performance. 

Requiring  the  supplier  to  provide  and  commit  to  the  processes  that  it  proposed  and  link  its  work 
products  to  the  IMS,  through  methods  described  in  Chapters  2  and  3,  facilitates  the  ability  to 
reference  its  processes  and  specified  work  products  as  part  of  its  activities  and  to  monitor  the 
completion  of  its  work  products  as  part  of  earned  value  management  (EVM)  reporting.  (See 
“Process  Monitoring  through  Artifact  Reviews”  on  page  58  in  Appendix  D  for  implementation 
details.) 

4.2  Process  Monitoring  Through  Review  of  Process  Metrics 

Supplier  organizations  should  regularly  gather  metrics  related  to  both  their  program  and  process 
performance  and  perform  periodic  audits  of  process  performance.  A  review  of  these  metrics  and 
audit  results  can  give  the  acquirer  needed  information  on  the  supplier’s  process  implementation 
and  performance.  (See  “Process  Monitoring  through  Review  of  Process  Metrics”  on  page  58  in 
Appendix  D  for  implementation  details.) 
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4.3  Process  Monitoring  Through  Process  Reviews 

Early  review  of  the  supplier  team’s  process  capability  reinforces  process  capability  in  low-risk 
teams  and  will  motivate  process  capability  in  high-risk  teams.  Whether  these  reviews  are  initiated 
by  the  acquirer  or  the  supplier,  ideally  they  include  participation  by  both. 

After  contract  award,  the  following  development  process  issues  are  common: 

•  Is  the  supplier  team’s  development  process  defined  and  being  implemented?  If  not,  when  will 
this  happen? 

•  Are  critical  development  processes  in  place  for  program  kick-off? 

•  Has  the  lead  supplier  put  a  plan  in  place  to  integrate  the  team’s  development  processes, 
including  all  subcontractors? 

The  acquirer  can  perform  process  reviews  through  the  access  clause  in  the  contract.  For  DoD 
acquisitions,  DCMA  can  perform  them  on  a  continuing  basis  as  part  of  its  contract  monitoring 
activities.  (See  “Process  Monitoring  through  Process  Reviews”  on  page  59  in  Appendix  D  for 
implementation  details.) 

4.4  Process  Monitoring  Using  Post-Award  Appraisai  Resuits 

Post-award  appraisal  results  (generally,  a  SCAMPI  Class  B  appraisal)  are  characterized  as 
strengths,  weaknesses,  or  gaps,  not  as  capability  levels  or  maturity  levels.  These  findings  provide 
an  accurate  indicator  of  the  state  of  process  areas  that  are  determined  to  be  critical  to  the  program. 

Process  areas  identified  as  low  risk  do  not  require  immediate  action,  but  continued  monitoring  is 
advised.  Those  identified  as  medium  risk  require  further  investigation  to  determine  what  is 
causing  implementation  gaps  and  what  steps  can  be  taken  to  improve.  Those  identified  as  high 
risk  require  immediate  action  as  they  pose  a  significant  barrier  to  successful  deployment  of 
critical  processes.  The  acquirer  can  collaborate  with  the  supplier  on  a  process  improvement  plan 
prioritized  according  to  the  appraisal  findings.  (See  “Process  Monitoring  through  Post-Award 
Appraisals”  on  page  60  in  Appendix  D  for  implementation  details.) 
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Appendix  A:  Interpreting  CMMI  Ratings 


Various  claims  made  about  capability  models  and  maturity  level  ratings  over  the  years  have  led  to 
some  misperceptions.  This  appendix  addresses  these  misperceptions.  Organizations  that  attain  a 
CMMI  maturity  level  rating  of  3  or  higher  should  have  documented  organizational  standards, 
defined  processes,  and  procedures  that  are  demonstrated  to  be  repeatable.  While  a  maturity  level  3 
supplier  should  tailor  the  organization’s  standard  processes  for  use  on  all  new  programs,  the 
acquirer  cannot  infer  that  the  processes  are  necessarily  effective  or  even  consistently  applied 
across  the  enterprise.  Some  warnings  for  the  acquirer  follow.  The  discussion  centers  on  maturity 
level  ratings  rather  than  process  capability  profiles.  This  is  because  the  majority  of  appraised 
organizations  end  up  with  a  maturity  level  rating,  and  maturity  levels  are  most  often  touted  by 
suppliers. 

A  CMMI  rating  or  CMMI  level  is  not  a  guarantee  of  program  sueeess. 

While  process  capability  and  organizational  maturity  have  been  shown  to  be  critical  factors  for 
program  success,  they  are  not  the  only  ones.  Mature  suppliers  that  have  capable  processes  in  place 
on  a  program  can  still  encounter  problems.  There  are  a  number  of  factors  that  contribute  to 
program  success  or  failure,  such  as  an  incomplete  understanding  of  the  job,  program  constraints 
(e.g.,  funding,  schedule),  the  skills  of  the  people  assigned  to  the  program,  and  the  maturity  of  the 
technology  applied  to  the  program.  Organizations  anticipate  that  adoption  of  CMMI-DEV  will 
contribute  to  program  success  because  it  is  an  improvement  over  previous  capability  maturity 
models  and  because  it  integrates  both  systems  and  software  engineering  in  a  single  framework 
that  instills  development  discipline.  However,  the  acquirer  cannot  assume  that  the  supplier  will 
execute  the  program  at  the  same  maturity  level  that  the  organization  has  achieved  and  advertised. 

Organizations  that  have  attained  CMMI  maturity  level  ratings  do  not  neeessarily  apply 
those  appraised  proeesses  to  a  new  program  at  program  start-up. 

Maturity  level  ratings  are  achieved  by  an  organizational  unit  (e.g.,  the  corporation,  a  major 
division,  a  few  key  programs)  and  are  not  an  indication  of  how  an  individual  program  is 
performing.  CMMI  maturity  level  ratings  indicate  potential  organizational  performance  and  how 
the  next  program  could  perform  based  on  a  sampling  of  past  and  current  programs  if  the  appraised 
processes  are  applied  in  the  same  manner  as  on  the  chosen  sample  programs.  While  organizations 
with  maturity  level  3  and  higher  maturity  level  ratings  are  expected  to  employ  their  organizational 
processes  on  new  programs,  there  is  evidence  that  some  organizations  do  not  or  that  it  takes  some 
time  before  processes  are  applied  to  new  programs.  Therefore,  acquirers  should  determine  if  rated 
organizations  have  a  policy  for  using  their  organizational  processes  on  new  programs  and  a 
history  of  following  that  policy.  CMMI-DEV,  VI. 3  contains  material  to  explicitly  address  process 
deployment  to  newly  started  programs  in  two  process  areas.  Organizational  Process  Focus  and 
Integrated  Project  Management. 
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Organizations  that  claim  CMMI  ratings  may  not  remain  committed  to  process 
improvement. 

While  most  organizations  that  invest  in  the  improvement  of  their  development  processes  take  the 
investment  seriously  and  commit  to  implementing  those  processes,  some  apparently  seek  a 
maturity  level  rating  only  for  use  in  pursuit  of  opportunities  (e.g.,  so  that  they  can  present  the 
credential  in  proposals).  These  organizations,  once  they  have  achieved  their  maturity  rating,  may 
not  apply  the  appraised  processes  as  implied. 

Additionally,  organizations  not  committed  to  process  improvement  may  choose  to  abandon  their 
processes  during  contract  execution  when  they  perceive  it  to  be  to  their  advantage  (e.g.,  costs  start 
to  overrun  or  schedule  slips  occur).  These  pressures  can  cause  the  acquirer  or  the  supplier  to 
abandon  mature  processes  by  not  performing  certain  design,  analysis,  or  verification  activities  in 
an  attempt  to  save  time  or  money.  Such  supplier  organizations  expose  acquirers  to  additional  risk 
through  poor  program  execution.  However,  there  are  ways  for  acquirers  to  identify  organizations 
that  are  not  committed  to  using  their  processes  as  a  normal  part  of  their  work.  For  example, 
organizations  that  are  actively  pursuing  process  improvement  have  a  demonstrated  change  history 
regarding  the  use  of  their  processes,  policies,  and  procedures.  Acquirers  can  request 
documentation  in  the  solicitation  that  provides  a  clear  indicator  of  active  use,  monitoring,  and 
improvement  of  these  artifacts.  Additionally,  suppliers  may  use  other  tools,  complementary  to 
CMMI,  that  provide  an  indication  of  their  commitment  to  process  improvement,  such  as  Lean  Six 
Sigma  or  Team  Software  Process®"^  (TSP^*^).^ 

CMMI  level  ratings  are  based  on  a  representative  sample  of  an  organization’s  programs 
and  may  not  refleet  performanee  on  all  programs. 

Ratings  apply  to  an  organizational  unit;  however,  it  is  often  not  practical  for  a  whole  organization 
to  be  appraised.  Historically,  organizations  identify  to  appraisal  teams  those  programs  that  are 
available  to  be  reviewed  in  the  upcoming  appraisal.  The  appraisal  team  selects  the  number  of 
programs  that  it  can  reasonably  appraise  in  the  allotted  time  and  that  it  deems  to  be  representative 
programs  of  the  organizational  unit  from  this  identified  set. 

Recent  changes  to  CMMI  are  intended  to  improve  program  selection  integrity  by  placing  more 
responsibility  on  lead  appraisers  to  ensure  a  representative  sample.  These  improvements  in  the 
ADS  seek  to  prevent  having  only  the  best  programs  be  chosen  (i.e.,  “cherry-picking”)  and 
provides  acquirers  with  a  better  understanding  of  the  breadth  of  the  appraisal.  These  changes  are 
not  reflected  in  earlier  versions  of  the  ADS. 

Most  organizations  are  far  from  being  homogeneous  entities.  In  fact,  many  defense  and 
commercial  suppliers  are  an  amalgamation  of  smaller  organizations  acquired  over  time.  They  are, 
therefore,  dispersed  geographically  and  bring  different  legacy  processes  to  the  table.  Even  though 
one  would  like  to  believe  that  all  sectors,  divisions,  and  locations  of  a  supplier  operate  in  the  same 
way,  this  is  seldom  the  case.  Consequently,  one  supplier  location  may  indeed  use  verified,  capable 
processes,  while  another  location  may  not.  Thus,  awarding  a  contract  to  a  supplier  at  location  A 
that  has  capable  processes  at  location  B  is  not  a  guarantee  that  the  capable  processes  will  be 
applied  to  the  program  at  location  A  immediately,  or  ever. 


For  more  information  on  continuous  process  improvement,  see  the  Continuous  Process  Improvement 
Transformation  Guidebook  [DoD  2006b]. 
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Organizations  that  claim  a  high  maturity  level  rating  (level  4  and  5)  are  not  necessarily 
better  suppliers  than  a  level  3  supplier. 

Maturity  levels  4  and  5,  when  compared  across  different  suppliers,  are  not  created  equal.  An 
organization  must  select  at  least  one  process  or  subprocess  that  is  important  to  its  business 
objectives  and  quantitatively  manage  (maturity  level  4)  or  optimize  (maturity  level  5)  it.  Another 
organization  would  likely  select  different  processes  or  subprocesses.  Thus,  if  the  chosen  processes 
or  subprocesses  are  not  those  of  value  to  the  current  program,  the  supplier’s  high  maturity  activity 
may  not  add  benefit.  Lead  appraisers  have  an  additional  responsibility  to  verify  that  the  chosen 
processes  relate  to  the  organization’s  business  objectives;  however,  acquisition  organizations 
should  have  insight  into  the  profile  of  capability  levels  across  all  the  process  areas  that  are  critical 
or  relevant  to  the  program.  Examination  of  the  capability  profile  clearly  shows  the  levels  of  any 
particular  process  area,  while  relying  on  a  maturity  level  will  not.  In  this  regard,  the  profile  is 
more  meaningful  than  relying  on  a  single  maturity  level. 

A  high  maturity  level  rating  is  not  an  indication  of  how  the  next  program  will  perform',  it  is  only 
an  indication  of  how  the  next  program  could  perform  provided  other  critical  success  factors 
remain  the  same.  Organizations  that  have  attained  high  maturity  level  ratings  are  not  immune  to 
the  aforementioned  pitfalls.  Achieving  a  high  maturity  rating  often  involves  modifying  the 
behavior  of  a  number  of  individuals  throughout  the  organization  and  dismpting  organizational 
inertia.  It  takes  management  focus  and  staff  commitment  to  improve.  Management  must  be 
diligent  to  assure  that  once  organizational  change  occurs,  the  behaviors  become  institutionalized. 
Clearly,  past  performance  only  indicates  the  potential  performance  for  the  current  program.  To 
achieve  the  anticipated  performance  on  the  current  program,  the  mature  organization  must 
instantiate  the  appraised  processes  on  that  program. 
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Appendix  B:  Questions  and  Checkiists 


Table  5:  Acquirer’s  Decision  Summary 


Selection 

Method  Title 

Responsibility 

Assignment 

Date 

Planned 

Implementation 

Date 

Actual 

Implementation 

Date 

Status 

Identify  the  Critical  Process  Areas  of  the  Program 

□ 

2.3.1  Self  Assessment 

□ 

2.3.2  Facilitated  Assessment 

Leverage  the  Process  Capabilities  of  the  Supplier 

□ 

3.1  Consider  Process  Proposals  in  Critical  Areas 

□ 

3.2  Consider  Integrated  Master  Plan  Documentation  of 

Proposed  Processes 

□ 

3.3  Consider  Incorporation  of  Process  Reviews  into  Proposed 

Program  Schedules 

□ 

3.4  Consider  the  Approach  to  Integration  with  Subcontractor 

Processes 

□ 

3.5  Consider  Approach  to  Integration  with  Acquirer  Processes 

□ 

3.6  Consider  Recent  Appraisal  Results 

□ 

3.7  Consider  Historical  Data  on  Process  Performance 
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c 

,o 

Responsibility 

Assignment 

Date 

Planned 

Implementation 

Date 

Actual 

Implementation 

Date 

Status 

o 

0) 

0) 

CO 

Method  Title 

□ 

3.8 

Consider  Recent  Post-Award  Appraisal  Data  form  Other 
Programs 

□ 

3.9 

Consider  Process  Compliance  Evaluations  During 

Execution 

□ 

3.10 

Consider  the  Reporting  of  Process  Statistics 

□ 

3.11 

Consider  Access  to  Selected  Process  Artifacts 

□ 

3.12 

Consider  Conducting  Pre-Award  Process  Appraisals 

□ 

3.13 

Consider  Conducting  Post-Award  Process  Appraisals 

□ 

3.14 

Consider  Process-Based,  Outcome-Oriented  Award  Fee 

Determinants 

Monitor  Supplier  Process  Performance  After  Contract  Award 

□ 

4.1 

Process  Monitoring  Through  Artifact  Reviews 

□ 

4.2 

Process  Monitoring  Through  Review  of  Process  Metrics 

□ 

4.3 

Process  Monitoring  Through  Process  Reviews 

□ 

4.4 

Process  Monitoring  Using  Post-Award  Appraisal  Results 
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Appendix  C:  Identify  the  Critical  Process  Areas  of  the 
Program 


The  two  methods  in  this  appendix  can  help  the  acquirer  assess  and  prioritize  the  program’s  critical 
process  areas.  These  methods  may  be  used  individually  or  in  combination.  When  the  acquirer 
chooses  the  method  to  be  deployed  on  the  program,  the  acquirer  staff  assigned  to  execute  this 
decision  may  then  use  this  detailed  implementation  guidance. 

SELF-ASSESSMENT  QUESTIONNAIRE 

The  questionnaire  in  Table  6  is  an  example  of  what  might  be  used  to  identify  the  process  areas 
that  are  important  to  the  program,  using  the  taxonomy  of  CMMI-DEV.  The  questionnaire  will 
also  work  well  in  a  spreadsheet.  By  identifying  those  process  areas  that  are  particularly  important, 
the  acquirer  can  focus  on  the  more  critical  processes  and  request  information  related  to  them.  This 
questionnaire  uses  the  specific  goals  of  each  process  area  in  CMMI-DEV  that  were  discussed  in 
Chapter  2. 

Table  6:  Process  Needs  Self-Assessment  Questionnaire 


Not  Important 

Slightly  Important 

Important 

Very  Important 

Essential 

# 

Process  Area  and  Goals 

1 

2 

3 

4 

5 

Project  Planning 

1. 

Establish  Estimates 

O 

O 

O 

O 

O 

2. 

Develop  a  Project  Plan 

O 

O 

o 

O 

O 

3. 

Obtain  Commitment  to  the  Plan 

O 

o 

o 

O 

o 

Project  Monitoring  and  Control 

4. 

Monitor  Project  Against  Plan 

O 

o 

o 

O 

o 

5. 

Manage  Corrective  Action  to  Closure 

O 

o 

o 

O 

o 

Supplier  Agreement  Management 

6. 

Establish  Supplier  Agreements 

O 

o 

o 

O 

o 

7. 

Satisfy  Supplier  Agreements 

O 

o 

o 

O 

o 
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Not  Important 

Slightly  Important 

Important 

Very  Important 

Essential 

# 

Process  Area  and  Goals 

1 

2 

3 

4 

5 

Integrated  Project  Management 

8. 

Use  the  Project’s  Defined  Process 

O 

O 

O 

O 

O 

9. 

Coordinate  and  Collaborate  with  Relevant 

Stakeholders 

O 

O 

o 

O 

O 

Risk  Management 

11. 

Prepare  for  Risk  Management 

O 

o 

o 

O 

o 

12. 

Identify  and  Analyze  Risks 

O 

o 

o 

O 

o 

13. 

Mitigate  Risks 

O 

o 

o 

O 

o 

Requirements  Management 

14. 

Manage  Requirements 

O 

o 

o 

O 

o 

Requirements  Development 

15. 

Develop  Customer  Requirements 

O 

o 

o 

O 

o 

16. 

Develop  Product  Requirements 

O 

o 

o 

O 

o 

17. 

Analyze  and  Validate  Requirements 

O 

o 

o 

O 

o 

Technical  Solution 

18. 

Select  Product  Component  Solutions 

O 

o 

o 

O 

o 

19. 

Develop  the  Design 

O 

o 

o 

O 

o 

20. 

Implement  the  Product  Design 

O 

o 

o 

O 

o 
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Not  Important 

Slightly  Important 

Important 

Very  Important 

Essential 

# 

Process  Area  and  Goals 

1 

2 

3 

4 

5 

Product  Integration 

21. 

Prepare  for  Product  Integration 

O 

O 

O 

O 

O 

22. 

Ensure  Interface  Compatibility 

O 

O 

o 

O 

O 

23. 

Assemble  Product  Components  and  Deliver  the 

Product 

O 

o 

o 

O 

o 

Verification 

24. 

Prepare  for  Verification 

O 

o 

o 

O 

o 

25. 

Perform  Peer  Reviews 

O 

o 

o 

O 

o 

26. 

Verify  Selected  Work  Products 

O 

o 

o 

O 

o 

Validation 

27. 

Prepare  for  Validation 

O 

o 

o 

O 

o 

28. 

Validate  Product  or  Product  Components 

O 

o 

o 

O 

o 

Configuration  Management 

29. 

Establish  Baselines 

O 

o 

o 

O 

o 

30. 

Track  and  Control  Changes 

O 

o 

o 

O 

o 

31. 

Establish  Integrity 

O 

o 

o 

O 

o 

Process  and  Product  Quality  Assurance 

32. 

Objectively  Evaluate  Processes  and  Work  Products 

O 

o 

o 

O 

o 

33. 

Provide  Objective  Insight 

O 

o 

o 

O 

o 

Measurement  and  Analysis 

34. 

Align  Measurement  and  Analysis  Activities 

O 

o 

o 

O 

o 

35. 

Provide  Measurement  Results 

O 

o 

o 

O 

o 
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Not  Important 

Slightly  Important 

Important 

Very  Important 

Essential 

# 

Process  Area  and  Goals 

1 

2 

3 

4 

5 

Decision  Anaiysis  and  Resoiution 

36. 

Evaluate  Alternatives 

O 

O 

O 

O 

O 

What  to  Do 

This  self-assessment  questionnaire  can  be  used  by  the  acquirer  in  the  context  of  the  following 
activities: 

•  Distribute  the  questionnaire  to  key  individuals  within  the  acquirer  staff,  and  ask  them  to 
independently  complete  it. 

•  Convene  a  meeting  of  all  responders. 

•  Discuss  the  evaluations  of  each  item  to  reach  a  consensus. 

•  Record  the  consensus  evaluation. 

FACILITATED  ASSESSMENT 

The  acquirer  can  conduct  a  facilitated  assessment  by  performing  the  following  activities: 

•  Review  program  documentation  to  develop  an  understanding  of  the  program  and  its  risk 
exposure. 

•  Interview  acquirer  staff  to  further  understand  the  program. 

•  Interview  program  stakeholders  to  develop  an  understanding  of  their  needs  and  expectations. 

The  results  of  the  assessment  are  delivered  in  a  report  identifying  and  prioritizing  the  process 
needs  of  the  program,  along  with  the  supporting  rationale. 

What  to  Do 

The  acquirer  arranges  for  a  facilitated  assessment  by  performing  the  following  steps: 

1 .  Contact  the  appropriate  organization  (e.g.,  a  qualified,  federally  funded  research  and 
development  center  [FFRDC],  such  as  the  SEI,  an  SEI  Partner  organization  qualified  to 
perform  appraisals,  or  the  various  software  engineering  centers  (SECs)  or  software  support 
activity  (SSA)  established  within  the  Services)  to  request  assistance  in  appraising  the 
process  needs  of  the  program. 
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2. 


Work  with  the  assigned  assessor*^  to  identify  documentation  needs,  knowledgeable  acquirer 
staff,  and  critical  stakeholders. 

3.  Provide  the  identified  documentation  to  the  assessor  for  review. 

4.  Work  with  the  assessor  to  schedule  interviews  with  the  acquirer  staff  and  critical 
stakeholders. 

When  to  Do  It 

The  selected  assessment  method'^  can  be  used  early  in  the  development  of  the  acquisition 
strategy. 

How  to  Use  the  Results 

Either  method  can  be  used  to  focus  on  the  process  areas  that  are  important,  very  important,  or 
essential  to  the  program  or  to  evaluate  the  process  capabilities  of  potential  offerors.  The  critical 
areas  identified  should  be  part  of  the  RFP  so  that  offerors  can  propose  processes  mapped  to  the 
critical  areas. 


The  facilitated  assessment  of  critical  process  areas  does  not  have  to  be  performed  using  a  SCAMPI  appraisal 
method;  therefore,  the  term  “assessor”  Is  used  here  Instead  of  “appraisal  lead.” 

The  term  “assessment”  as  used  here  means  that  the  acquirer  hires  someone  from  the  outside  to  determine  the 
critical  process  areas.  No  methodology  or  standard  is  implied. 
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Appendix  D:  Leverage  the  Process  Capabilities  of  the 
Supplier 


This  appendix  provides  implementation  details  for  the  various  approaches  discussed  in  Chapter  3. 
It  covers  various  ways  to  get  a  chosen  approach  into  a  contract  with  a  supplier. 

Traditionally,  for  DoD  acquisitions,  all  proposal  information  is  requested  as  part  of  Section  L  of 
the  RFP  and  the  evaluation  of  the  resulting  proposal  submission  is  governed  under  the  provisions 
of  Section  M.  In  cases  where  the  organizations  expected  to  participate  in  the  acquisition  are 
known  entities  and  the  risks  of  process  implementation  are  relatively  well  understood,  this 
traditional  approach  works  well. 

An  alternative  approach,  especially  when  the  potential  offerors  are  not  as  well  known,  is  to 
request  the  process-related  information  as  part  of  a  request  for  information  (RFI)  prior  to  the 
release  of  an  RFP.  This  approach  gains  access  to  the  process-related  information  prior  to  release 
of  the  RFP  so  additional  conditions  might  be  included  in  the  RFP  to  further  reduce  program  risk. 
For  example,  if  an  offeror  is  not  able  to  adequately  demonstrate  that  it  has  adequate  processes  in 
place  and  is  not  involved  in  a  process  improvement  program,  the  risk  to  the  program  may  be  great 
enough  to  require  an  acquisition-related  process  appraisal  as  part  of  the  acquisition  process. 

While  this  alternative  requires  additional  time,  it  may  be  the  only  way  to  gather  information  on 
process  capability  for  organizations  that  have  not  been  previously  appraised.  It  also  provides  some 
additional  risk  quantification  when  used  to  supplement  the  findings  from  a  previously  performed 
third-party  appraisal.  If  the  acquirer  believes  there  is  enough  process-related  risk  to  the  program, 
an  appraisal  could  be  performed  on  an  offeror  prior  to  the  RFP  release  (e.g.,  as  part  of  an  advisory 
multi-step  process  prior  to  RFP  release  [Federal  Acquisition  Regulation  Subpart  15.202]).  If  an 
additional  (unanticipated)  offeror  submits  a  proposal,  this  new  offeror  would  be  subjected  to  the 
same  requirements  as  the  other  offerors  (i.e.,  the  appraiser  would  conduct  the  appraisal  after 
receipt  of  the  proposal). 

Guidance  is  also  provided  for  approaches  to  take  when  process-related  risks  associated  with  the 
potential  winner  are  discovered.  Assuming  evaluation  notices  (ENs)  have  been  issued  and 
discussions  have  begun,  the  potential  winner  can  be  requested  to  address  its  weaknesses  in  its  best 
and  final  offer  (BAFO)  and  include  corrections  in  its  final  proposal  document  (FPD).  This  course 
of  action  can  be  considered  when  the  risks  identified  are  severe  enough  to  potentially  increase  the 
award  price.  However,  when  such  action  is  warranted,  particularly  on  programs  in  which  process 
capability  is  deemed  to  be  especially  critical,  having  the  opportunity  to  mitigate  risk  in  advance  of 
award  has  its  benefits. 

Once  the  acquirer  chooses  the  methods  to  be  deployed  on  the  program,  the  acquirer  staff  assigned 
to  execute  these  decisions  may  then  refer  to  the  detailed  implementation  guidance  in  this  appendix 
to  help  them  implement  the  methods. 
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3.1  CONSIDER  PROCESS  PROPOSALS  IN  CRITICAL  AREAS 


The  acquirer  should  be  aware  that  requesting  process  information  can  result  in  large  volumes  of 
information  from  each  offeror.  By  requesting  process  information  the  acquirer  can  rapidly 
accumulate  proposal  page  counts  that  are  daunting  to  review,  and  the  sheer  volume  of  information 
may  overshadow  other  key  aspects  of  the  proposal  that  are  just  as  important  to  evaluate.  The 
acquirer  should  concentrate  on  the  areas  of  critical  concern  by  carefully  selecting  the  information 
needed  and  ensuring  that  it  directly  supports  evaluation  and  decision  making.  Any  process 
information  requested  can  be  tailored  by  the  offeror  for  application  to  the  program.  This  amount 
of  activity  is  charged  to  the  offeror  bid  and  proposal  budget,  so  the  acquirer  should  select  the 
process  areas  so  the  amount  of  work  does  not  unnecessarily  drive  up  the  offeror’s  costs. 

Information  on  how  the  offeror  plans  to  tailor  and  deploy  processes  on  the  program  can  be 
provided  in  several  ways  and  to  varying  levels  of  detail.  If  the  acquirer  is  familiar  with  the  offeror 
organization  that  will  be  performing  on  the  program  or  if  a  pre-award  appraisal  has  been 
conducted,  the  appraisal  findings  and  the  offeror  schedule  for  training  and  deployment  of  the 
processes  may  be  sufficient. 

When  the  acquirer  is  less  familiar  with  the  offeror  and  its  process  execution  history,  more  detailed 
information  may  be  necessary.  With  a  focus  on  the  critical  process  areas  for  the  program,  the 
acquirer  should  select  the  types  of  information  required.  This  information  can  be  provided  in  the 
form  of  process  proposals,  which  should  be  the  offeror’s  organizational  processes  tailored 
specifically  for  the  program. 

The  offeror  should  also  provide  process  information  on  major  subcontractors.  Typically,  prime 
contractors  take  two  approaches  with  their  subcontractors: 

1 .  have  subcontractors  execute  the  prime  contractor’s  processes  as  team  members 

2.  have  subcontractors  execute  their  own  processes  in  development  of  a  product  to  be 
delivered  to  the  prime 

The  acquirer  should  exercise  caution  in  cases  in  which  prime  contractors  integrate  their  processes 
with  those  of  their  subcontractors.  This  method  may  take  many  months  to  result  in  efficient,  well 
understood,  and  executed  processes  for  the  program.  (See  “Consider  the  Approach  to  Integration 
with  Subcontractor  Processes”  on  page  44  in  Appendix  C.)  The  offeror  should  be  required  to 
describe  its  approach  and  provide  process,  schedule,  and  task  information  (i.e.,  process  proposals, 
integrated  master  plan  (IMP),  integrated  master  schedule  (IMS),  work  breakdown  structure 
(WBS),  and  plan  for  training  all  team  members)  in  its  proposals.  At  a  minimum,  the  offeror 
should  be  required  to  provide  process  documentation  for  critical  program  process  areas  for  all 
major  subcontractors,  especially  those  that  will  be  executing  their  responsibilities  under  the 
contract  using  their  own  processes.  Depending  on  the  risks  associated  with  the  subcontractor 
portion  of  the  work,  process  proposals  may  also  be  required. 

If  the  offeror  is  using  process  methodologies  other  than  CMMI  and  SCAMPI  (e.g.,  ISO/IEC 
15288,  12207,  15504;  ISO  9001,  EIA  632,  and  IEEE  1220),  or  if  there  are  processes  to  be 
implemented  that  are  not  captured  by  CMMI-DEV  (e.g.,  manufacturing  practices),  the  acquirer 
should  require  the  offeror  to  map  the  information  into  a  format  consistent  with  the  one  being  used 
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to  map  processes  to  CMMI-DEV  (as  shown  in  Chapter  2  and  Appendix  B).  This  mapping  should 
include  process  descriptions  and  tailoring  as  well  as  any  formal  audit  results. 

180-9001:2008  is  a  quality  management  standard  for  development  created  and  maintained  by  the 
International  Organisation  for  Standardisation  (ISO).  The  American  National  Standard  equivalent 
is  ANSI/ISO/ASQ  Q9001-2008.  Organizations  attain  180-9001:2008  registration  through  an 
independent  audit  process  that  examines  their  compliance  to  identified  practices.  Because  ISO- 
9001:2008  is  primarily  a  quality  standard,  it  is  less  prescriptive  of  the  development  process  than 
CMMI-DEV,  but  it  does  provide  evidence  of  a  process  focus  within  the  organization.  If  the 
bidding  organization  (prime  contractor  or  subcontractors)  claims  ISO  9001:2008  registration,  the 
acquirer  should  determine  if  the  processes  being  bid  are  included.  Since  ISO  9001:2008  is  a 
quality  system  registration,  it  only  includes  those  processes  that  are  specifically  cited.  Therefore,  a 
development  organization  may  have  a  subset  of  its  processes  (e.g.,  assembly  and  test  of  hardware) 
within  its  registration  but  may  not  include  its  full  set  of  development  processes. 

The  information  supplied  in  offeror  process  proposals  can  be  formally  evaluated  and  scored  as  a 
factor  in  source  selection. 

What  to  Do 

The  acquirer  can  request  descriptions  of  the  processes  in  areas  critical  to  the  program.  These  must 
then  be  examined  in  detail  to  thoroughly  assess  the  potential  risk  of  execution.  These  requests  can 
be  made  in  Section  L  of  the  REP.  The  acquirer  should  include  evaluation  criteria  related  to 
process  performance  and  risk  in  Section  M  of  the  REP.  Any  process  proposal  information 
provided  by  the  offeror  should  be  above  and  beyond  what  is  included  in  the  software  development 
plan  (SDP)  or  systems  engineering  plan  (SEP),  which  will  have  much  of  the  information  already 
provided  for  evaluation  [DoD  2011b].  The  descriptions  can  include  a  list  of  work  products 
produced  by  the  process,  as  well  as  some  samples  of  these  work  products. 

When  to  Do  It 

Collecting  process  proposals  is  the  first  step  in  ensuring  that  the  proposed  processes  will  be 
actually  applied  to  the  program.  If  there  are  concerns  over  the  detailed  implementation  of 
processes  in  specific  areas,  additional  information  can  be  requested.  As  part  of  the  proposal 
evaluation,  the  acquirer  can  then  determine  the  capability  and  suitability  of  these  processes  for  the 
program. 

If  the  acquirer  knows  the  expected  offerors  well,  and  they  have  demonstrated  the  application  of 
capable  processes  to  other  similar  programs,  the  information  can  be  requested  in  the  REP.  If  the 
acquirer  does  not  know  the  expected  offerors  well,  the  information  can  be  in  an  REI  for  earlier 
pre-qualification  evaluation. 

How  to  Use  the  Results 

Process  proposals  form  the  basis  for  determining  the  risk  related  to  their  execution  on  the  program 
and  provide  the  basis  for  requiring  adherence  to  those  processes  as  part  of  the  contract.  The 
acquirer  should  request  that  each  prime  contractor  and  major  subcontractor  provide  descriptions 
of  the  processes  that  they  commit  to  use  immediately  after  contract  award. 
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One  method  of  evaluating  these  proposed  processes  is  to  use  a  SCAMPI  Class  C  appraisal  prior  to 
award.  This  appraisal  evaluates  the  process  proposals  in  the  critical  program  process  areas  for 
compliance  with  CMMI-DEV.  Because  it  looks  only  at  the  process  proposals  to  determine  the 
intent  of  the  organization,  this  appraisal  method  is  considerably  less  intmsive,  less  time 
consuming,  and  less  expensive  than  more  comprehensive  appraisals. 

Process  proposals  are  appraised  to  answer  the  following  questions: 

•  Do  the  proposed  processes  reflect  the  best  practices  of  CMMI-DEV? 

•  Are  the  processes,  particularly  those  in  the  process  areas  identified  as  critical  to  program 
success,  complete  and  adequate  to  perform  the  desired  functions? 

•  Are  the  activities  and  work  products  listed  in  the  processes  visible  in  the  proposed  IMS? 

The  result  of  this  appraisal  is  a  collection  of  risks  arising  from  gaps  between  CMMI-DEV  and  the 
proposed  processes.  These  risks  may  be  factored  into  the  source  selection  criteria.  These  process 
proposals  can  also  be  referenced  in  the  contract  of  the  selected  supplier,  thereby  providing  a 
contractual  commitment  to  execute  the  processes  on  the  program,  as  proposed. 

Commitment  to  follow  a  set  of  processes  can  provide  the  acquirer  with  the  means  to  monitor 
supplier  performance  after  award.  Having  process  descriptions,  including  definitions  of  work 
products,  can  also  provide  the  acquirer  with  the  means  to  verify  that  the  supplier  intends  to 
execute  the  processes. 

3.2  CONSIDER  INTEGRATED  MASTER  PLAN  DOCUMENTATION  OF  PROPOSED 
PROCESSES 

What  to  Do 

The  offeror  should  identify  processes  that  map  to  critical  process  areas  identified  for  the  program 
in  the  REP  and  include  references  to  organizational  process  materials  (or  materials  already 
tailored  for  the  program)  in  the  IMP.  The  offeror  should  also  include  dates  for  development 
program  team  training  (if  any)  and  process  deployment  in  the  IMS.  The  offeror  should  be  required 
to  describe  its  approach  and  provide  process,  schedule,  and  task  information  (i.e.,  process 
proposals,  IMP,  IMS,  work  breakdown  structure,  and  plan  for  the  training  of  team  members)  in  its 
proposals.  These  activities  should  be  traceable  to  the  work  breakdown  structure  as  referenced  in 
the  IMP  and  IMS.  A  matrix  can  be  provided  for  offerors  to  map  their  proposed  processes  to  the 
program’s  identified  critical  process  areas.  In  addition,  offerors  should  map  the  artifacts  of  their 
critical  processes  to  the  IMS  and  identify  the  nomenclature  they  are  using  to  identify  those 
artifacts. 

If  the  acquirer  is  familiar  with  the  offeror  organization  that  will  perform  on  the  program  or  if  a 
pre-award  appraisal  has  been  conducted,  the  formal  appraisal  findings  and  the  offeror  schedule  for 
training  and  deployment  of  the  processes  may  be  sufficient  to  provide  an  understanding  of  the 
maturity  of  each  offeror’s  set  of  processes. 
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When  to  Do  It 


Normally,  the  requirement  to  provide  an  IMP,  IMS,  a  program,  work  breakdown  structure,  and 
the  associated  basis  of  estimate  are  included  in  the  RFP  along  with  directions  to  complete  the 
process  portion  of  the  IMP  and  IMS  and  any  mapping  matrix. 

How  to  Use  the  Results 

When  in  receipt  of  the  above  information,  the  acquirer  should  determine  where  the  offeror 
identifies  its  program-critical  processes  and  how  it  identifies  the  artifacts  for  those  critical 
processes  in  its  IMS.  The  completeness  of  the  offeror’s  process  implementation  can  also  be 
examined  in  its  process  mapping  against  the  program’s  critical  process  areas.  The  completeness  of 
the  submission  along  with  the  degree  of  support  provided  by  the  offeror’s  basis  of  estimates  can 
allow  the  acquirer  to  estimate  the  degree  to  which  each  offeror  is  planning  to  execute  its  processes 
on  the  program. 

3.3  CONSIDER  INCORPORATION  OF  PROCESS  REVIEWS  INTO  PROPOSED 
PROGRAM  SCHEDULES 

What  to  Do 

The  acquirer  should  request  that  the  offeror  include  planned  process  reviews  in  its  IMP  and  IMS 
and  request  that  the  offeror  provide  a  list  of  the  process  work  products  for  the  program’s  critical 
process  areas  that  will  be  included  in  the  following: 

•  earned  value  management  system  (if  applicable) 

•  integrated  baseline  review  (if  applicable) 

•  data  accession  list  (DAL) 

If  the  acquirer  includes  a  request  for  process  reviews  in  the  IMS,  the  acquirer  should  include  this 
requirement  in  the  model  contract  and  in  the  final  contract  so  the  results  can  be  monitored.  If  an 
offeror  has  not  fully  incorporated  process  reviews  into  its  proposed  IMS,  the  acquirer  may  want  to 
direct  them  to  include  additional  reviews  in  the  supplier’s  final  proposal  submission.  If  the 
acquirer  does  not  include  a  request  for  process  reviews  in  the  IMS,  the  acquirer  can  still  negotiate 
the  inclusion  of  the  reviews  in  the  final  IMS  as  a  risk  reduction  activity  and  introduce  the  subject 
as  part  of  a  review  of  the  IMS  in  discussions. 

When  to  Do  It 

For  DoD  acquisitions,  in  Section  L  of  the  RFP,  the  acquirer  should  include  directions  to 
incorporate  proposed  process  reviews  in  the  IMP  and  IMS  and  include  process  work  products  in 
the  IMS.  Based  on  the  risks  to  the  process,  determined  earlier,  the  acquirer  should  ask  for  the 
appropriate  process  reviews  to  be  included  immediately  after  contract  award  and  before  the  initial 
review  of  the  IMP  and  IMS. 

How  to  Use  the  Results 

The  acquirer  should  answer  the  following  questions  in  reviewing  the  offerors’  proposals: 
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1 .  Has  the  offeror  incorporated  appropriate  levels  of  process  reviews  in  its  IMP  and  IMS 
submittals? 

2.  Is  the  first  process  review  soon  enough  after  award  to  allow  the  acquirer  to  determine  if  the 
critical,  early  processes  are  being  employed  effectively? 

3.  Does  the  offeror  propose  appropriate  metrics  that  will  allow  the  acquirer  and  its 
management  to  verify  and  track  the  rollout  of  the  offeror’s  processes? 

4.  Are  the  key  artifacts  and  work  products  required  by  offeror  processes  identified  and  tracked 
in  the  IMS? 

The  IBR  establishes  the  EVMS  baseline.  A  review  of  process  deployment  could  assess  whether 
the  deployed  processes  are  included  in  support  of  the  baseline,  particularly  through  the  listing  of 
process  work  products  as  milestone  events. 

3.4  CONSIDER  THE  APPROACH  TO  INTEGRATION  WITH  SUBCONTRACTOR 
PROCESSES 

Typically,  prime  contractors  take  two  approaches  with  their  subcontractors: 

1 .  Have  the  subcontractors  execute  the  prime  contractor’s  processes  as  team  members. 

2.  Have  the  subcontractors  execute  their  own  processes  in  development  of  a  product  to  be 
delivered  to  the  prime  contractor. 

The  acquirer  should  exercise  caution  in  cases  where  prime  contractors  integrate  their  processes 
with  those  of  their  subcontractors,  as  this  method  may  take  many  months  to  result  in  efficient, 
well-understood,  and  executed  processes  for  the  program. 

What  to  Do 

The  acquirer  should  request  each  offeror’s  approach  to  the  integration  of  its  critical  processes  with 
its  subcontractors,  including  the  means  that  the  offeror  proposes  to  eliminate  risks  associated  with 
start-up  and  training.  This  approach  provides  an  indication  of  how  well  this  important  aspect  of 
process  integration  has  been  thought  out.  When  available,  the  offeror  can  provide  an  ADS  or 
appraisal  findings  for  all  subcontractors,  especially  those  executing  their  responsibilities  under  the 
contract  using  their  own  processes.  Depending  on  the  risks  associated  with  the  subcontractor 
portion  of  the  work,  process  proposals  may  also  be  required 

When  to  Do  It 

This  information  is  important  for  any  program  that  includes  a  significant  amount  of 
subcontracting.  If  the  expected  offerors  are  well  known  and  have  demonstrated  the  application  of 
mature  processes  to  other  similar  programs,  this  information  can  be  requested  in  the  RFP.  If  the 
expected  offerors  are  not  well  known,  this  information  can  be  requested  in  an  RFI  for  earlier  pre¬ 
qualification  evaluation. 

How  to  Use  the  Results 

There  are  two  different  situations  that  result  in  two  different  approaches  to  address  the  potential 
risk  of  prime  contractor  and  subcontractor  process  relationships.  Both  can  occur  on  the  same 
acquisition  program  within  different  bidding  teams. 
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1 .  If  a  prime  contractor  proposes  that  all  subcontractors,  or  even  some  of  the  critical 
subcontractors  will  use  their  own  processes  to  perform  development,  then  the  evaluation 
team  must  be  prepared  to  determine  whether  the  prime  and  the  subcontractors  have 
substantiated  the  following  about  themselves: 

-  They  are  sufficiently  mature  in  their  own  right  to  execute  their  processes  effectively. 

-  They  have  addressed  in  their  proposal  how  they  will  integrate  their  processes  to 
minimize  duplication  and  effectively  address  all  the  practices  related  to  the  program’s 
critical  processes. 

Unless  the  bidding  team  has  already  analyzed  and  agreed  on  process  interfaces,  a 
significant  risk  exists  that  the  processes  will  not  effectively  execute  from  the  outset. 
Considering  that  early  execution  of  time-critical  processes  most  often  determines  the 
success  of  a  program,  it  is  imperative  that  the  bidding  team  address  those  interfaces  in 
detail  prior  to  contract  award  and  assure  the  evaluation  team  that  it  has  an  effective 
approach. 

2.  If  the  prime  contractor  proposes  a  distributed  development  facility  with  each  major 
subcontractor  using  its  own  processes,  the  risk  resides  in  the  ability  of  the  prime  to  deploy 
an  integrated  development  environment  (IDE)  and  to  integrate  its  overarching  processes 
with  the  subcontractors  through  an  organizational  and  technical  interface  structure.  If  the 
prime  intends  to  execute  a  distributed  development  environment,  its  productivity  will  be 
lower  at  the  start  of  the  program  unless  it  plans  to  implement  the  IDE  prior  to  award  and 
train  its  subcontractors  on  the  tools  that  it  intends  to  use. 

If  the  prime  contractor  proposes  a  single  development  site  with  all  subcontractors  using  the 
prime’s  processes,  the  risk  resides  in  the  ability  of  the  prime  to  rapidly  train  and  employ  its 
subcontractors.  If  the  prime  intends  to  execute  this  training  after  the  award,  its  productivity  will  be 
lower  at  the  start  of  the  program  and  early  design  and  architectural  activities  may  incur  additional 
risk  and  effort. 

If  a  prime  contractor  proposes  that  it  will  impose  its  processes  on  subcontractors,  then  it  should 
have  a  plan  for  training  the  subcontractors  on  its  organizational  processes.  If  the  prime  contractor 
proposes  to  do  so  after  award,  then  it  is  imposing  considerable  risk  that  the  early  activities  will  not 
effectively  use  its  processes.  It  is  not  unreasonable  to  expect  a  prime  contractor  to  propose 
instituting  training  of  prospective  subcontractors  prior  to  contract  award.  If  such  training  is  not 
proposed,  then  the  evaluation  team  can  consider  the  early  execution  of  program  processes  to  be  at 
risk,  at  least  at  risk  for  errors  or  omissions. 

In  assessing  the  risk  to  the  program,  the  acquirer  should  determine  whether  the  approaches  chosen 
by  each  offeror  will  add  to  or  mitigate  the  risks  associated  with  effective  program  start-up  or  with 
implementing  effective  processes  critical  to  the  program. 

3.5  CONSIDER  THE  APPROACH  TO  INTEGRATION  WITH  ACQUIRER  PROCESSES 
What  to  Do 

The  acquirer  should  request  each  offeror’s  approach  to  integrating  critical  program  processes  that 
will  be  executed  in  coordination  with  the  acquirer.  This  information  indicates  how  well  each 
offeror  can  address  critical  program  processes  in  concert  with  the  acquirer  processes  after  award. 
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When  to  Do  It 


This  information  is  important  for  any  program  that  includes  a  significant  dependence  on  process 
capability.  If  the  expected  offerors  are  well  known  and  have  demonstrated  the  application  of 
mature  processes  to  other  similar  programs,  this  information  can  be  requested  in  the  RFP.  If  the 
expected  offerors  are  not  well  known,  this  information  can  be  requested  in  an  RFI  for  earlier  pre¬ 
qualification  evaluation. 

How  to  Use  the  Results 

A  detailed  imderstanding  of  the  acquirer  processes,  and  plans  for  developing  and  improving  those 
processes,  is  a  necessary  first  step.  If  otherwise  appropriate,  it  is  a  good  idea  for  the  acquirer  to 
provide  the  offeror  with  a  brief  description  (or  list)  of  the  processes  it  expects  to  put  in  place  so 
that  the  offerors  can  provide  inputs  or  suggest  additional  processes  that  might  further  reduce  risk 
in  the  overall  program. 

Offerors  can  provide  detailed  benefits  and  approaches  to  acquirer  and  supplier  process  integration. 
The  acquirer  might  consider  the  following  typical  attributes  of  process  integration  as  beneficial: 

•  increased  visibility  into  supplier  operations 

•  enhanced  timeliness  for  the  delivery  or  sharing  of  program  management  information 

•  enhanced  accuracy  and  understanding  of  information 

•  enhanced  ability  of  both  the  acquirer  and  supplier  to  make  (specific  and  appropriate)  program 
decisions 

•  enhanced  plans  for  the  acquirer’s  participation  with  the  supplier  in  integrated  project  teams 
(IPTs)  (Preferably,  such  participation  is  not  limited  to  information  sharing,  but  includes 
shared  decision-making.) 

Typically  the  acquirer  should  expect  a  supplier  to  recommend  the  integration  of  risk  management, 
requirements  management,  requirements  development,  and  configuration  management  processes. 
In  certain  circumstances,  it  might  also  be  appropriate  for  the  acquirer  to  consider  the  integration 
of  project  monitoring  and  control,  measurement  and  analysis,  and  product  integration  (if  the 
acquirer  will  act  as  the  final  or  system  integrator)  processes. 

To  be  considered  viable,  offeror  recommendations  for  process  integration  should  be  defined  well 
enough  to  be  clearly  implemented  in  a  timely  manner.  In  general,  recommendations  for  process 
integration  should  also  address  known  or  identifiable  program  risks. 

3.6  CONSIDER  RECENT  APPRAISAL  RESULTS 

When  available,  data  on  previous  organizational  appraisals  can  be  gathered  from  each  prime 
contractor  and  subcontractor  that  will  perform  major  development  activity. 

With  the  release  of  version  1.2  of  the  SCAMPI  Method  Definition  Document  (MDD),  the  ADS 
must  contain  the  following  information: 

•  the  appraisal  sponsor  and  sponsor’s  organizational  affiliation 

•  the  appraisal  team  leader,  appraisal  team  members,  and  their  organizational  affiliations 
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•  the  organizational  unit  appraised 

•  the  projects  and  other  groups,  their  function  and  placement  within  the  organizational  unit,  and 
their  geographic  locations 

•  sample  size,  in  quantifiable  terms  of  the  organizational  sample  in  relation  to  the  size  of  the 
organizational  unit 

•  CMMl  model  used  (version,  representation,  and  disciplines) 

•  appraisal  method  used  (name  and  version) 

•  itemization  of  process  areas  rated  and  process  areas  not  rated 

•  maturity  level  and  capability  level  ratings  assigned 

•  goal  ratings  for  process  areas  within  the  scope  of  the  appraisal 

•  dates  of  on-site  activity 

•  date  the  ADS  was  issued  and  period  of  the  validity  of  the  appraisal  results 

•  statement  affirming  that  all  SCAMPI  requirements  were  met 

•  signature  of  appraisal  team  leader  and  sponsor  with  indication  of  agreement  to  publish 
appraisal  results 

The  SCAMPI  ADS  Example  Template,  in  appendix  A  of  the  SCAMPI  V1.2  MDD,  contains  the 
guidance  provided  to  lead  appraisers  on  the  content  of  the  VI. 2  ADS  [SEl  2006b].  There  is  a 
requirement  to  define  the  percentage  of  people  and  projects  appraised  in  relation  to  the 
organizational  unit.  There  is  also  a  requirement  to  provide  the  percentage  of  each  critical  factor 
identified  in  appraisal  planning  covered  by  the  organizational  scope  in  relation  to  the 
organizational  unit. 

Examples  of  critical  factors  include  the  following: 

•  application  domains  (i.e.,  lines  of  business) 

•  geographical  breadth 

•  disciplines  (i.e.,  systems  engineering,  software  engineering,  hardware  engineering) 

•  effort  types  (e.g.,  development,  maintenance,  services) 

•  project  types  (e.g.,  legacy,  new  development) 

•  customer  type  (e.g.,  commercial,  DoD,  NASA) 

•  lifecycle  models  in  use  within  the  organization  (e.g.,  spiral,  evolutionary,  waterfall, 
incremental) 

With  the  release  of  the  SCAMPI  VI. 3  MDD  in  March  2011,  these  factors  have  been  described  as 
“sampling  factors”  to  enable  more  cost-effective,  reliable  sampling  of  the  appraised 
organization’s  process  performance.  This  change  should  reduce  earlier  acquisition  concerns  that 
only  selected  (“focus”)  projects  are  actually  performing  to  the  level  documented  in  the  appraisal. 

The  acquirer  can  use  this  information  to  determine  the  relative  sample  size  of  the  projects 
appraised  as  part  of  the  ADS  documentation. 

The  appraisal  findings  usually  take  the  form  of  a  briefing  containing  an  overview  of  strengths  and 
weaknesses  and  the  final  results  of  the  appraisal  described  in  the  ADS.  Often,  individual  practice 
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characterizations  aggregated  to  the  organizational  level  are  also  presented.  (At  the  request  of  the 
appraisal  sponsor,  program-level  data  may  also  be  included  in  the  final  findings,  but  this  may  not 
be  found  in  the  ADS.) 

What  to  Do 

For  the  prime  contractor  and  for  each  subcontractor  or  team  member  involved  in  major 
development  activity,  request  the  ADS  and  the  appraisal  findings  that  were  generated  as  the  result 
of  any  SCAMPI  Class  A  appraisals  within  the  last  three  years.  This  documentation  can  be 
requested  for  subcontractors  as  well  as  for  the  prime.  The  most  reliable  data  are  from  the  most 
recent  appraisal.  Any  supplier  claiming  a  maturity  level  in  its  proposal  should  be  willing  to 
release  this  documentation. 

The  acquirer  can  also  request  the  offeror  to  specify  the  exact  relationship  of  the  organization  that 
was  appraised  to  the  bidding  organization.  Was  the  bidding  organization,  and  specifically  projects 
within  the  bidding  organization,  within  the  scope  of  that  appraisal?  If  not,  will  the  bidding 
organization  be  using  the  processes  that  were  within  the  scope  of  the  appraisal? 

Determining  whether  or  not  the  organization’s  ADS  represents  the  organization  as  a  whole  and 
whether  or  not  it  represents  the  team  executing  the  program  is  important.  Methods  used  to  sample 
the  organization  are  varied  and  in  the  past  have  not  been  governed  by  any  strict  rales  or  disclosure 
requirements.  Appraisals  performed  using  SCAMPI  VI  .3  employ  more  robust  sampling 
approaches  than  earlier  versions  demanded. 

The  acquirer  should  request  information  on  the  number  of  active  development  projects  at  the  site 
being  proposed  to  perform  the  work,  the  number  of  active  development  projects  contained  in  the 
bidding  organization  (if  different),  and  the  number  of  active  development  projects  contained  in 
the  organization  that  was  appraised  (if  different). 

Each  offeror  (prime  contractor  and  major  subcontractor)  that  provides  ADS  data  can  also  be 
required  to  provide  information  on  the  number  of  major  development  projects  that  are  currently 
under  contract  in  the  appraised  organization.  Additionally,  the  acquirer  should  request  the  number 
of  staff  members  assigned  and  the  revenue  for  each  project  (e.g.,  major  development  programs 
may  be  defined  as  those  greater  than  18  months  in  duration  and  greater  than  $10  million  in  total 
contract  size). 

When  reviewing  the  ADS,  the  acquirer  should  determine  the  number  and  identities  of  the 
programs  appraised  in  the  ADS  and  compare  them  to  the  active  major  development  program 
totals  provided.  This  determination  should  provide  a  good  estimate  of  whether  the  sampling  of  the 
organization  was  reasonable  and  representative.  It  will  also  provide  some  idea  of  the  sampling 
rates  that  the  bidding  organizations  used  for  their  appraisals.  For  more  recent  appraisals,  this 
information  will  be  included  within  the  ADS.  Regardless,  it  is  prudent  for  the  acquirer  to  pursue 
this  issue  to  ensure  that  sampling  used  by  the  appraised  organization  is  understood. 

When  to  Do  It 

By  requesting  this  information  in  an  RFI  preceding  the  formal  RFP  release,  the  acquirer  can 
examine  the  potential  offerors’  organizational  maturity  and  process  capability.  This  information 
can  then  help  fine-tune  the  evaluation  criteria.  Issuing  such  an  RFI  also  puts  potential  offerors  on 
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notice  that  process  capability  is  a  concern.  Such  notice  may  discourage  offerors  that  lack  a 
process  focus  from  participating  in  the  RFP,  or  may  encourage  them  to  take  significant  action  to 
rectify  their  shortfalls. 

Offerors  can  be  instructed  (in  Section  L  of  the  RFP)  to  provide  this  information  as  part  of  their 
proposal  submissions.  Evaluation  criteria  can  be  included  in  Section  M  of  the  RFP.  Phrase  these 
evaluation  criteria  in  the  form  of  risk  assessments — brisks  posed  to  the  program  as  a  result  of  the 
process  proficiency  of  offeror  organizations  especially  in  the  process  areas  of  key  importance  to 
the  program.  Special  consideration  can  be  given  to  the  process  areas  that  must  be  enabled  from 
the  start  of  the  program,  as  performance  in  the  early  stages  of  the  program  is  critical  to  overall 
success. 

How  to  Use  the  Results 

The  acquirer  should  use  the  data  supplied  in  the  ADS  to  address  the  following  questions: 

1.  Is  the  organization  named  in  the  ADS  the  same  organization  that  is  bidding  on  and  will 
execute  the  program? 

If  not,  ask 

a.  Is  the  organization  named  in  the  ADS  at  a  higher  level  in  the  organizational  structure 
than  the  organization  bidding  on  and  planning  to  execute  the  program? 

•  If  not,  investigate  the  relationship  between  the  two  organizations.  If  the  relationship 
is  tenuous,  the  information  in  the  ADS  is  probably  not  useful. 

•  If  so,  then  investigate  the  current  relationship  between  the  two  organizations.  Has  the 
organization  been  reorganized  since  the  appraisal?  If  so,  there  could  be  some 
differences  in  the  organizational  commitment  to  the  processes  to  be  employed  that 
must  be  verified 

2.  How  many  projects  were  included  in  the  appraisal?  (Use  some  of  the  supplemental 
information  to  support  this  analysis.) 

a.  What  was  the  size  of  the  organization  stated  in  the  scope? 

b.  What  criteria  were  used  to  select  programs  to  be  included  in  the  appraisal? 

c.  How  many  development  programs  are  currently  active  in  that  organization? 

d.  What  is  the  size  of  the  bidding  organization  where  the  work  will  be  performed? 

3.  Was  the  appraisal  performed  less  than  two  years  ago? 

It  is  unlikely  that  an  organization  that  is  actively  involved  in  process  improvement  would 
leave  all  its  processes  untouched  for  more  than  two  years.  In  organizations  at  maturity  or 
capability  level  3  and  higher  (and  occasionally  at  lower  levels),  process  assets  should  be 
gathered  in  an  organizational  repository  and  be  placed  under  change  control.  Change  records 
from  this  repository  are  a  clear  indicator  of  process  change.  Review  of  these  records  provides 
two  pieces  of  information: 

a.  an  indication  of  the  activity  in  the  process  repository.  A  noticeable  absence  of 
submissions  or  revisions  may  be  an  indication  of  a  lack  of  process  focus 

b.  a  view  into  the  changes  in  the  processes  since  the  last  appraisal.  Substantial  changes  in 
the  processes  may  invalidate  the  findings  of  prior  appraisals.  Explanations  and 
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motivations  for  the  changes,  as  well  as  discussions  of  the  applicability  of  the  prior 
appraisals  can  be  requested. 

4.  Which  process  areas  were  included  in  the  appraisal?  Organizations  can  choose  to  appraise 
using  one  of  two  model  representations,  staged  or  continuous.  Staged  appraisals  require 
certain  sets  of  process  areas  to  be  appraised  to  achieve  a  maturity  level  rating  and  success  is 
designated  by  an  indication  that  the  goals  of  the  process  area  are  “satisfied.”  Continuous 
appraisals  require  that  each  process  area  included  in  the  appraisal  be  examined  and  a 
capability  level  be  designated  for  each  process  area.  When  viewing  the  results  of  an 
appraisal,  it  is  important  to  determine  if  the  specific  process  areas  that  were  designated  as 
critical  to  the  program  were  included  in  the  appraisal. 

a.  If  SAM  was  not  included  in  the  appraisal,  the  prime’s  process  for  dealing  with  critical 
subcontractors  has  not  been  appraised  and  the  risk  associated  with  managing 
subcontractor  activities  should  be  examined  in  more  detail. 

Use  the  appraisal  findings  to  answer  the  following  questions: 

5.  Do  the  appraisal  findings  declare  Supplier  Agreement  Management  (SAM)  process  area  not 
applicable?  If  SAM  is  related  to  critical  program  processes,  then  there  would  be  additional 
risk  to  the  program  in  selecting  an  organization  that  has  not  demonstrated  capability  in  a 
critical  area. 

6.  Will  weaknesses  noted  in  the  appraisal  findings  have  a  significant  impact  on  the  program? 
Has  the  organization  addressed  or  corrected  any  of  these  noted  weaknesses?  Has  the 
sufficiency  of  the  corrections  been  evaluated?  Appraisal  findings  are  required  to  list  any 
critical  weaknesses  found.  Critical  weaknesses  are  those  that  keep  a  maturity  or  capability 
level  from  being  achieved  and  will  be  clearly  indicated  in  the  findings.  Identified  strengths 
and  weaknesses,  especially  critical  weaknesses,  can  be  evaluated  against  the  program’s 
required  process  areas,  particularly  when  they  are  identified  in  the  portion  of  the 
organization  that  would  be  performing  on  the  program. 

Use  the  publicly  reported  appraisal  data  to  corroborate  the  ADS  information  on  the  PARS  [SEI  2]. 
If  discrepancies  are  found  between  the  publicly  reported  data  and  the  data  supplied  in  the  proposal 
response,  additional  requests  for  information  may  be  required. 

3.7  CONSIDER  HISTORICAL  DATA  ON  PROCESS  PERFORMANCE 

Reviewing  the  results  of  past  appraisals  is  only  one  way  to  examine  the  history  of  process 
application  within  an  organization.  The  appraisal  process  provides  an  in-depth  examination  of  the 
process  utilization  on  a  selected  sample  of  projects  either  completed  or  in-process.  However, 
because  of  the  intrusive  nature  of  the  appraisal  process,  it  is  not  practical  to  examine  more  than  a 
few  programs  within  an  organization.  While  the  chosen  programs  are  expected  to  be 
representative  of  the  broader  organization,  they  may  constitute  an  extremely  small  sample,  and  it 
may  be  impossible  to  verify  their  application  throughout  the  remainder  of  the  organization. 

An  alternative  method  of  examining  the  history  of  process  application  within  an  organization  is  to 
review  that  organization’s  own  records  of  process  application.  Many  higher-maturity 
organizations  collect  data  on  the  application  and  performance  of  their  processes. 
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This  data  typically  includes  metrics  that  track  the  following: 

•  the  organizational  processes  applied  to  a  program 

•  modifications  to  the  organizational  processes  required  for  a  program 

•  compliance  of  program  activities  with  the  applied  processes 

If  this  data  is  available,  it  can  be  valuable  to  provide  insight  into  the  process  focus  of  the 
organization,  the  consistency  of  process  application,  and  the  monitoring  and  enforcement  of 
processes. 

Most  solicitations  request  examples  of  performance  on  past  contracts.  Looking  for  correlations 
between  the  examples  provided  in  the  supplier’s  proposal  and  projects  participating  in  prior 
process  appraisals  can  also  be  useful. 

What  to  Do 

Request  information  detailing  the  historical  use  of  organizational  processes  on  past  and  current 
projects.  The  acquirer  should  seek  statistics  on 

•  the  breadth  of  process  application  across  the  organization  (e.g.,  what  percentage  of  projects 
employ  standard  or  tailored  versions  of  the  organization’s  standard  processes) 

•  the  consistency  of  process  application  (e.g.,  what  percentage  of  processes  applied  to  projects 
are  modified  from  the  organization’s  standard  processes) 

•  project  compliance  with  the  defined  processes  (e.g.,  metrics  resulting  from  process 
monitoring  and  process  audits) 

When  to  Do  It 

The  optimal  time  to  request  this  information  is  in  the  formal  RFP.  If  the  acquirer  has  a  detailed 
understanding  of  the  process  capabilities  of  potential  offerors,  it  can  understand  the  potential  risk 
of  selecting  an  offeror  with  immature  development  processes.  An  analysis  of  the  information 
provided  allows  the  acquirer  to  structure  the  risk  evaluation  during  technical  evaluation  and 
source  selection. 

If  appropriate  to  the  evaluation,  the  acquirer  can  request  that  the  offerors  share  information  about 
which  of  the  programs  that  were  evaluated  in  past  performance  evaluations  also  have  historical 
process  data. 

How  to  Use  the  Results 

The  historical  data  about  the  process  performance  of  programs  in  the  same  (or  similar)  domain  as 
the  program  being  bid  is  more  relevant  and  likely  to  be  a  better  predictor  of  potential 
performance. 

For  consistency  of  process  application  (percentage  of  processes  actually  applied  to  projects)  it  is 
better  if  all  or  most  programs  in  the  organization  apply  tailored  versions  of  the  standard  processes. 
Historical  data  showing  process  capability  for  programs  at  multiple  points  in  their  lifecycle  is  a 
clear  indicator  of  an  organization’s  commitment  to  the  implementation  of  standard  processes  in 
the  development  of  their  work  products. 
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3.8  CONSIDER  RECENT  POST-AWARD  APPRAISAL  DATA  FROM  OTHER 
PROGRAMS 

What  to  Do 

The  acquirer  should  request  all  available  post-award  appraisal  information  on  any  recent  (i.e., 
within  two  years)  post-award  appraisals  of  the  bidding  organization  and  sites  proposed  for 
development  activities. 

When  to  Do  It 

The  optimal  time  to  request  this  information  is  in  the  formal  RFP,  but  this  information  may  be 
helpful  at  any  time  during  the  life  of  the  program. 

How  to  Use  the  Results 

Acquiring  organizations  have  often  requested  post-award  appraisals  of  winning  organizations.  If 
the  prime  contractor  or  key  development  subcontractors  have  had  a  post-award  appraisal  on  a 
newly  awarded  program,  the  acquirer  should  use  the  results  of  these  appraisals  to  answer  the 
following  questions: 

1 .  Has  the  post-award  appraisal  result  shown  that  the  organization  adopts  its  processes  rapidly 
and  effectively? 

2.  Has  there  been  more  than  one  instance  of  post-award  appraisals  on  a  single  project?  Did  the 
sequence  of  appraisals  demonstrate  progress  in  resolving  process  deficiencies? 

3.  Did  the  results  indicate  effective  or  ineffective? 

3.9  CONSIDER  PROCESS  COMPLIANCE  EVALUATIONS  DURING  EXECUTION 

Some  organizations  have  process  compliance  evaluations  as  part  of  regular  product  and  process 
quality  assurance  reviews  and  others  make  those  evaluations  part  of  a  more  global  Mission 
Assurance  review  process.  Regardless  of  how  the  individual  organization  performs  that  function, 
it  benefits  the  acquirer  to  have  access  to  the  process  reviews  performed  on  the  specific  program 
for  which  it  is  responsible.  Most  process-focused  organizations  welcome  customer  involvement  in 
their  process  programs,  primarily  because  it  improves  understanding  of  the  process  issues  that 
improve  program  performance. 

What  to  Do 

For  DoD  acquisitions,  when  preparing  the  RFP,  the  acquirer  should  include  a  requirement  in 
“Instructions  for  Preparation  of  the  Offeror’s  Proposal”  in  Section  L  to  confirm  agreement  to  the 
acquirer’s  participation  in  process  compliance  evaluation  actions  and  request  the  offeror  to 
propose  how  that  participation  would  best  fit  its  organizational  process  enforcement  approach. 

Typically,  DCMA  relies  on  the  FAR  standard  inspection  (access)  clause  to  gain  access  to  the 
contract,  supplier,  products,  services,  data,  and  other  things  it  needs.  This  approach  is  sufficient 
for  many  contract  monitoring  purposes.  If  the  acquirer  intends  to  have  a  support  party,  such  as 
DCMA  or  an  FFRDC,  be  involved  in  verifying  process  compliance,  that  intention  can  be 
identified  in  the  RFP  so  agreements  can  be  made  regarding  access  to  information  by  the  support 
party  prior  to  award. 
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When  to  Do  It 


The  acquirer  can  lay  the  groundwork  for  participation  in  the  process  compliance  evaluations  as 
part  of  the  RFP.  By  making  the  request  when  competition  is  paramount,  the  offerors  should  be 
more  willing  to  meet  the  request.  An  additional  option  is  to  evaluate  the  offeror’s  response  to  the 
request,  include  it  in  any  ENs  created,  and  resolve  any  issues  as  part  of  the  discussion  process. 

How  to  Use  the  Results 

During  the  proposal  evaluation,  if  the  acquirer  is  proactive,  it  can  use  the  willingness  of  an  offeror 
to  ensure  that  processes  are  executed  on  the  new  program  and  to  provide  visibility  during 
execution  as  an  indication  of  lower  risk  and  a  higher  probability  of  success. 

3.10  CONSIDER  THE  REPORTING  OF  PROCESS  STATISTICS 
What  to  Do 

The  acquirer  may  simply  request  that  the  supplier  furnish  the  process  metrics  that  are  collected  for 
the  program.  If  the  supplier’s  response  is  inadequate,  additional  metrics  can  be  requested.  For 
example,  some  or  all  of  the  common  process  statistics  listed  below  may  be  applicable  to  the 
program;  however,  the  acquirer  should  consider  the  supplier’s  effort  needed  for  reporting  of  these 
metrics: 

•  process  implementation  timelines  (i.e.,  how  long  from  project  kick-off  did  it  take  to 
implement  and  begin  using  a  process) 

•  number  (or  percent)  of  the  organization’s  standard  processes  being  adopted  (tailored  for  use) 
by  the  project 

•  number  (or  percent)  of  work  products  required  by  a  process  that  were  actually  produced 

•  number  (or  percent)  of  supplier  processes  actually  reviewed  during  or  after  execution  for  the 
purpose  of  identifying  problems,  providing  lessons  learned,  or  for  improving  the  process 

•  number  of  process  defects  identified  over  the  project  lifecycle 

•  percent  of  process  defects  that  were  corrected  (closed) 

•  number  of  project  processes  found  to  be  compliant  with  CMMI-DEV  (at  each  capability 
level) 

•  number  of  process  audits  performed  on  the  developer’s  processes,  for  each  process 

•  number  of  improvements  made  to  the  developer’s  processes,  for  each  process 

When  to  Do  It 

The  reporting  of  process  metrics  or  statistics  can  be  requested  in  the  RFP,  or,  if  omitted,  may  be 
easily  requested  before  contract  award. 

How  to  Use  the  Results 

A  mature,  well-defined,  and  understandable  set  of  process  metrics  and  statistics  is  desirable.  The 
timeliness  of  process  metrics  can  support  continuous  insight  into  the  supplier’s  process  capability 
and  may  also  provide  insight  into  its  process  readiness  to  support  major  program  milestones. 
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The  technical  means  and  timelines  associated  with  providing  process  metrics  to  the  acquirer 
should  be  well  defined  and  realistic.  A  combination  of  independent  process  metrics  into  key 
process  indicators  is  helpful,  as  are  process  metrics  that  measure  the  adequacy  or  performance  of 
integration  with  the  acquirer  processes,  where  appropriate.  Process  control  metrics  should  be 
found  in  high  maturity  organizations  (i.e.,  those  with  maturity  level  ratings  of  4  or  5).  Failure  of 
the  supplier  to  provide  meaningful  process  metrics  and  statistics  increases  program  risk.  Ideally, 
the  supplier  has  a  process  for  developing  metrics  and  then  using  them  for  specific  periods  or  when 
triggered  by  certain  events  instead  of  simply  applying  the  same  set  of  metrics  to  each  program. 

3.1 1  CONSIDER  ACCESS  TO  SELECTED  PROCESS  ARTIFACTS 

Process  performance  can  be  monitored  through  the  evaluation  of  the  artifacts  produced  by  those 
processes  when  applied  to  the  program.  The  artifacts  needed  for  process  performance  evaluation 
are  accessible  to  the  acquirer  by  doing  one  or  more  of  the  following: 

•  reference  them  in  the  IMS  and/or  IMP 

•  reference  them  on  the  DAL 

•  reference  them  on  the  CDRL 

•  participate  in  integrated  process  teams  with  the  supplier 

•  attend  peer  reviews  or  other  program  reviews 

Process  implementation  indicators  (Plls),  which  suppliers  may  collect  for  their  own  SCAMPI 
appraisals,  can  be  used  if  they  are  from  the  actual  program. 

What  to  Do 

The  acquirer  should  request  that  the  supplier  illustrate  process  performance  by  identifying 
artifacts  produced  through  the  execution  of  the  program  processes,  including  the  process 
descriptions  for  the  critical  process  areas  of  the  program.  The  acquirer  then  should  review  the 
process  documents  to  identify  work  products  that  may  be  useful  in  assessing  process  performance. 
Periodically,  the  acquirer  should  obtain  copies  of  the  identified  work  products  from  the  supplier 
and  analyze  them  to  ascertain  the  degree  of  deployment  of  processes  on  the  program. 

When  to  Do  It 

The  RFP  can  establish  the  requirements  for  the  offerors  to  delineate  their  process  descriptions  and 
provide  access  to  relevant  work  products.  Access  to  the  work  products  can  be  addressed  through 
the  CDRL  and  the  DAL. 

Process  artifacts  can  be  monitored  throughout  the  duration  of  the  acquisition.  In  the  course  of  the 
acquisition,  different  process  areas  vary  in  their  level  of  importance.  When  reviewing  artifacts  for 
process  performance,  the  acquirer  should  choose  the  processes  that  are  most  significant  at  the 
current  time  given  the  phase  of  the  program. 

How  to  Use  the  Results 

If  the  supplier  is  contractually  obligated  to  perform  specific  processes,  the  acquirer  should  first 
assess  the  impact  of  any  non-compliance  and  address  the  non-compliance  issue  with  the  supplier 
to  determine  any  justification  for  the  discrepancy.  If  reasonable  justification  does  not  exist  and  the 
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impact  on  the  program  is  significant,  the  acquirer  should  first  try  to  address  the  issue  informally 
with  the  supplier,  ensuring  the  understanding  of  both  the  requirements  of  the  contract  and  the 
acquirer’s  expectations.  If  this  does  not  resolve  the  issue,  the  acquirer  should  consider  directing 
the  supplier  to  comply  with  stated  contractual  requirements  using  formal  contracting  actions. 

3.12  CONSIDER  CONDUCTING  PRE-AWARD  PROCESS  APPRAISALS 
What  to  Do 

The  process  appraisals  referenced  in  this  guidebook  are  adapted  (tailored)  to  the  situations  most 
useful  to  acquisition,  generally  limited  to  SCAMPI  Class  B  and  Class  C  appraisals.  A  SCAMPI 
Class  B  appraisal  team  may  be  as  small  as  two  or  the  investigation  team  may  be  expanded  to 
include  other  stakeholders  such  as  representatives  of  the  user  community,  supplier  process 
engineers,  or  other  affected  organizations.  The  design  of  the  appraisal  allows  for  a  focused 
productive  dialog  structured  by  the  relevant  process  areas  in  CMMf-DEV.  A  SCAMPI  Class  C 
appraisal  may  be  conducted  by  one  or  more  qualified  and  trained  individuals.  The  output  of  either 
of  these  appraisal  methods  is  a  list  of  findings  from  the  appraisal  and  a  list  of  risks  that  these 
findings  pose  for  the  project. 

In  general,  if  the  acquirer  desires  to  perform  an  appraisal,  it  does  not  need  appraisal  expertise;  the 
acquirer  can  work  with  a  trained  and  qualified  appraisal  lead  to  conduct  the  type  of  appraisal 
desired.  The  appraisal  lead  works  with  the  sponsor  to  clarify  the  needs  and  requirements  for  the 
appraisal.  The  appraisal  lead  may  then  recommend  various  approaches  to  fulfill  the  requirements. 
Once  an  approach  is  chosen,  the  appraisal  lead  begins  to  develop  detailed  plans  and  estimates  for 
approval  by  the  sponsor. 

To  employ  these  appraisals,  the  acquirer  should  use  the  following  key  steps: 

1 .  Determine  that  an  appraisal  is  warranted. 

2.  Work  with  a  qualified  appraisal  lead  to  do  the  following: 

a.  Define  the  goals  and  objectives  of  the  appraisal.  An  experienced  appraisal  lead  can  be 
helpful  in  ensuring  that  the  design  and  execution  of  the  appraisal  delivers  the  expected 
information  and  value. 

b.  Develop  a  detailed  appraisal  plan.  The  appraisal  lead  works  with  the  sponsor  to 
translate  the  goals  of  the  appraisal  into  a  detailed  appraisal  plan  with  estimates  of  time 
and  resources. 

c.  Determine  the  size  of  the  appraisal  team;  appraisals  typically  require  an  appraisal  team 
(i.e.,  individuals  in  addition  to  the  appraisal  lead).  The  team  size  and  qualifications  of 
the  team  members  have  an  impact  on  the  time,  cost,  and  depth  of  the  investigation. 
Team  members  can  include  acquirer,  user,  supplier/offeror,  or  external  personnel.  The 
appraisal  lead  works  with  the  sponsor  and  project  stakeholders  to  develop  a  staffing 
strategy  appropriate  to  the  type  of  appraisal  and  the  goals  of  the  sponsor. 

d.  Be  clear  about  the  outputs  of  the  appraisal  and  how  they  will  be  used  to  satisfy  the  goals 
of  the  sponsor. 

3.  Approve  the  appraisal  plan. 

4.  Arrange  for  appraisal  resources. 
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5.  Help  coordinate  with  appraised  entities. 

6.  Analyze  the  data. 

The  appraisal  team  collects  data  from  the  organization.  This  data  can  be  documents,  work 
products,  processes,  written  affirmations,  and  interviews.  The  appraisal  team  uses  CMMI-DEV  as 
a  framework  for  collecting  and  evaluating  the  evidence  provided  by  the  organization. 

Typical  outputs  from  the  appraisal  include  the  following: 

•  statements  of  strengths  and  weaknesses  mapped  to  model  practices 

•  practice  characterizations,  typically  risk  or  level  of  compliance 

•  other  information  requested  by  the  sponsor  or  deemed  important  by  the  team 

When  to  Do  It 

These  on-site  appraisals  must  be  scheduled  to  support  source  selection  needs  within  the  planned 
selection  period.  Planning  for  the  use  of  these  methods  often  precedes  the  release  of  the  RFP,  and 
depends  on  how  the  results  of  the  appraisals  are  fed  into  the  source  selection  process.  If  results  of 
the  appraisals  are  an  independent  risk  factor,  for  example,  timing  of  the  appraisals  may  be 
flexible.  But  if  the  results  of  the  appraisals  feed  other  teams,  the  appraisals  may  need  to  be 
scheduled  relatively  early  in  the  source  selection  period. 

How  to  Use  the  Results 

Frequently,  appraisal  results  are  included  in  a  broader  look  at  elements  such  as  contractor 
performance  assessment  reports.  This  broader  look  may  require  careful  timing  of  appraisal  visits 
so  that  the  appraisal  information  can  be  integrated  with  other  risk  determinants. 

3.13  CONSIDER  CONDUCTING  POST-AWARD  PROCESS  APPRAISALS 
What  to  Do 

In  the  RFP  package,  the  acquirer  should  reserve  the  right  to  execute  post-award  appraisals  using 
CMMI-DEV  focused  on  the  process  areas  that  are  determined  to  be  critical  to  the  program  at  any 
given  time.  The  acquirer  should  not  choose  CMMI  maturity  levels  as  the  scope  of  post-award 
appraisals;  instead,  it  should  follow  the  direction  given  in  this  guidebook  and  scope  any  appraisal 
to  those  process  areas  deemed  critical  and  proven  to  be  beneficial  to  the  success  of  the  program. 

At  the  appropriate  time,  the  acquirer  should  execute  a  SCAMPI  Class  B  appraisal  using  an 
authorized  SCAMPI  Class  B  appraisal  lead  and  a  team  trained  and  qualified  according  to  the 
SCAMPI  Class  B  appraisal  method.  Depending  on  the  model  and  organizational  scope,  this  type 
of  appraisal  can  be  completed  in  five  to  seven  working  days,  with  only  a  portion  of  this  time  at  the 
offeror’s  site. 

The  acquirer  should  make  sure  that  this  requirement  extends  to  subcontractors  so  that  the  entire 
supplier  team  may  be  appraised. 

When  to  Do  It 

Ideally,  it  is  best  to  reserve  the  option  for  post-award  appraisals  prior  to  RFP  release.  The  acquirer 
should  include  information  on  the  time  frame  of  the  planned  post-award  appraisals  and  provide  a 
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range  during  which  the  appraisals  could  be  planned  and  require  that  the  offeror  include  its 
proposed  dates  in  its  IMS. 

When  planning  the  second  post-award  appraisal,  it  may  be  appropriate  for  the  acquirer  to  schedule 
it  preceding  significant  milestones.  The  importance  of  different  processes  varies  throughout  the 
development  lifecycle.  Just  as  requirements  development,  requirements  management,  and  project 
planning  may  dominate  process  attention  in  the  early  phases,  increased  attention  to  configuration 
management,  verification,  and  validation  may  merit  greater  attention  later  in  the  lifecycle. 

How  to  Use  the  Results 

See  Section  4.4  on  page  28  and  its  associated  appendix. 

3.14  CONSIDER  PROCESS-BASED,  OUTCOME-ORIENTED  AWARD  FEE 
DETERMINANTS 

To  facilitate  discussion  and  share  proven  incentive  strategies  across  the  entire  U.S.  Defense 
acquisition  workforce,  the  DoD  established  the  award  and  incentive  fees  community  of  practice 
(COP)  under  the  leadership  of  the  Defense  Acquisition  University  (DAU)  [DoD  2007].  The  COP 
serves  as  the  repository  for  all  acquisition-related  materials,  including  policy  information,  training 
courses,  examples  of  good  award  fee  arrangements,  and  other  supporting  resources. 

What  to  Do 

An  approved  acquisition  strategy  may  result  in  award  fees  being  included  in  the  approved 
contract.  Process  improvement  planning  and  execution  can  be  part  of  award  fee  determination. 

The  acquirer  should  evaluate  effective  process  implementation  across  the  supplier  team  to 
mitigate  integration  risks.  An  evaluation  plan,  linked  to  award  fee  timing,  must  be  developed  and 
coordinated  with  stakeholders.  For  DoD  acquisitions,  agencies  such  as  DCMA  can  assist  with 
such  efforts. 

When  to  Do  It 

The  first  decision  point  is  associated  with  assuring  that  the  acquisition  strategy  allows  award  fees. 
If  it  does,  the  best  time  to  introduce  process  discipline  as  an  award  fee  element  is  in  the  RFP. 

How  to  Use  the  Results 

Award  fees  give  the  acquirer  a  powerful  way  to  recognize  achievement  or  failure  in  risk 
mitigation.  The  results  provide  an  excellent  method  to  encourage  team  accomplishment  through  a 
reward  system.  It  provides  regular  checks  for  continued  progress  and  management  attention.  (See 
Appendix  B,  “Systems  Engineering  in  an  Award  Fee  Plan,”  in  the  Guide  for  Integrating  Systems 
Engineering  into  DoD  Acquisition  Contracts  [DoD  2006d],  for  further  guidance.) 
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Appendix  E:  Monitoring  Suppiier  Process  Performance  After 
Contract  Award 

4.1  PROCESS  MONITORING  THROUGH  ARTIFACT  REVIEWS 
What  to  Do 

Joint  reviews  of  progress  are  frequent  and  familiar  to  most  acquirers.  Occasional  reviews  of  the 
process  quality  assurance  data  provide  excellent  insight  into  areas  that  may  pose  risks  to 
development.  Some  organizations  establish  separate  reviews  for  technical  performance,  cost  and 
schedule  (i.e.,  EVMS)  performance,  and  development  process  performance. 

When  to  Do  It 

Link  process  reviews  with  other  related  activities,  such  as  ISO  audits  or  the  company’s 
independent  process-improvement  appraisals. 

How  to  Use  the  Results 

The  acquirer  should  use  the  results  to  manage  risks  to  ensure  successful  completion  of  the 
development  program. 

4.2  PROCESS  MONITORING  THROUGH  REVIEW  OF  PROCESS  METRICS 
What  to  Do 

Process  performance  is  a  measure  of  the  results  achieved  by  following  a  process.  These  measures 
are  used  to  determine  if  processes  are  performing  within  the  expected  bounds  set  by  the 
organization.  The  acquirer  should  request  documentation  from  the  contracted  organization  that 
includes  its  established  performance  baselines  and  performance  models  for  the  processes  critical 
to  the  program.  This  documentation  should  be  available  in  any  high  maturity  organization..  Also, 
throughout  the  life  of  the  program,  the  acquirer  should  request  performance  data  periodically,  as 
documented  by  the  supplier,  to  effectively  monitor  these  processes. 

When  to  Do  It 

Monitoring  process  performance  metrics  can  be  addressed  in  the  RFP.  Examples  of  process 
performance  monitoring  can  be  examined  in  pre-award  SCAMPI  appraisals  if  properly  scoped. 
Shortly  after  contract  award,  the  following  can  be  agreed  to: 

•  the  exact  process  performance  metrics 

•  the  expected  amount  of  time  before  specific  metrics  would  be  available 

•  the  frequency  that  metrics  will  be  reported 

Monitoring  these  metrics  can  occur  throughout  the  life  of  the  program,  prioritized  based  on  risk  if 
necessary,  to  properly  manage  the  acquirer’s  resources. 
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How  to  Use  the  Results 


Based  on  the  supplied  performance  baselines,  the  need  for  supplier  action  should  be  evident  when 
metrics  fall  outside  the  expected  bounds  of  performance.  The  need  for  supplier  action  is  most 
evident  for  suppliers  using  statistical  process  control.  Both  the  acquirer  and  supplier  can  jointly 
review  thresholds  and  confirm  that  they  are  understood  and  appropriate.  The  acquirer  should 
monitor  and  track  these  actions  to  ensure  they  come  to  appropriate  closure. 

4.3  PROCESS  MONITORING  THROUGH  PROCESS  REVIEWS 

The  acquirer  should  obtain  process  performance  reports  from  the  supplier  to  gain  insight  into  the 
gaps  that  the  supplier  has  identified  in  its  own  processes.  If  these  are  not  available,  the  acquirer 
should  consider  conducting  process  reviews. 

What  to  Do 

For  high-risk  efforts,  or  if  there  are  doubts  about  the  supplier  team’s  development  processes,  a 
high-level  evaluation  of  the  supplier  team’s  defined  processes  can  be  performed  within  a  few 
weeks  of  contract  award.  The  evaluation  provides  direct  information  concerning  the  intent  of  the 
supplier  team  to  fully  define  its  development  processes  and  integrate  them  into  a  cohesive 
development  process.  If  the  supplier  does  not  pay  attention  to  these  activities,  they  can  likely 
cause  program  failure.  A  SCAMPI  Class  C  appraisal  is  an  effective,  quick,  low-cost  method  for 
this  type  of  evaluation.  The  initial  evaluation  can  be  followed  by  a  second,  more  in-depth 
evaluation  of  the  integrated  development  process  capability.  The  timing  of  this  second  evaluation 
depends  on  the  situation,  and  the  SCAMPI  Class  B  appraisal  is  of  significant  value  when  more 
formality  is  needed  in  these  cases. 

In  low-risk  efforts,  the  acquirer  may  elect  to  perform  just  the  in-depth  evaluation  at  an  appropriate 
time — ^perhaps  two  months  or  so  after  contract  award.  In  both  the  high  and  low  risk  cases,  the 
acquirer  may  choose  to  perform  follow-on  evaluations  to  ensure  supplier  process  weaknesses  are 
addressed  appropriately  and  there  is  no  process  performance  degradation  over  time. 

When  to  Do  It 

The  acquirer  conducts  one  review  shortly  after  contract  award,  when  the  initial  processes  have 
been  tailored  and  are  in  execution,  to  obtain  a  benchmark  of  process  performance.  They  can 
conduct  a  second  review  later  in  the  program,  possibly  around  the  same  time  as  a  major  program 
decision  point  (e.g.,  critical  design  review). 

In  addition  to  reviews,  the  acquirer  may  request  process  metrics  and  the  results  of  supplier  quality 
audits  of  its  development  processes  or  products. 

How  to  Use  the  Results 

The  acquirer  should  use  the  results  of  process  reviews,  process  audits,  or  process  metrics  to  ensure 
that  the  supplier’s  development  process  is  mature,  capable,  and  responsive  to  acquirer  needs.  Any 
process  interfaces  to  acquirer  processes  should  be  analyzed  and  understood.  Process  review  data 
should  provide  a  picture  of  firm  commitment  by  the  supplier  to  process  capability  and  continuous 
improvement.  Process  gaps  and  findings  can  be  considered  for  their  risk  implications.  The  action 
plans  that  are  part  of  risk  mitigation  can  then  be  tracked  to  resolution. 
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4.4  PROCESS  MONITORING  THROUGH  POST-AWARD  APPRAISALS 


What  to  Do 

The  acquirer  should  provide  instructions  on  approximate  time  frames  for  the  conduct  of  post¬ 
award  such  appraisals.  In  general,  two  process  appraisals  are  beneficial.  The  acquirer  can  then 
schedule  and  conduct  an  appraisal  of  the  supplier  team. 

To  employ  this  method,  it  is  not  necessary  to  have  appraisal  expertise.  Most  often,  the  acquirer 
can  work  with  an  appraisal  lead  trained  and  qualified  to  conduct  the  appraisal  desired.  The 
appraisal  lead  works  with  the  sponsor  to  clarify  the  needs  and  requirements  for  the  appraisal.  The 
appraisal  lead  may  then  recommend  alternative  approaches  to  fulfill  the  requirements.  Once  an 
approach  is  selected,  the  appraisal  lead  develops  detailed  plans  and  estimates  for  approval  by  the 
sponsor  (i.e.,  the  acquirer). 

The  acquirer  should  be  careful  when  choosing  an  appraisal  lead.  Many  suppliers  employ  internal 
staff  who  are  trained  in  CMMI  and  are  qualified  to  lead  appraisals.  Although  many  appraisal  leads 
also  offer  process  improvement  consulting  services  to  help  client  organizations  achieve  desired 
process  maturity,  it  is  preferable  to  avoid  using  appraisal  leads  employed  by  suppliers,  either 
directly  or  as  consultants.  While  such  appraisal  leads  offer  the  advantage  of  familiarity  with  the 
organizations  being  appraised,  this  advantage  is  offset  by  the  potential  conflict  of  interest.  It  is 
best  for  the  acquirer  to  choose  an  appraisal  lead  who  is  independent  from  the  supplier 
organization  and  has  experience  within  the  technology  domain  of  the  project. 

The  acquirer  should  clearly  communicate  the  goals  to  the  appraisal  lead  and  ensure  that  the 
appraisal  lead  understands  the  following: 

•  the  process  areas  that  are  critical  to  the  project 

•  the  scope  of  the  appraisal  (unlike  most  appraisals  that  examine  the  processes  within  an 
organization,  this  appraisal  examines  processes  only  within  the  program) 

•  the  current  status  of  the  program 

•  the  motivation  for  the  appraisal  (e.g.,  to  encourage  rapid  process  application  on  a  newer 
project,  to  check  for  compliance  commitments  made  during  bidding,  or  to  assess  problem 
areas  within  the  program) 

•  the  expectations  for  reporting  (e.g.,  a  report  of  process  capability  in  selected  process  areas,  a 
report  of  process  strengths  and  weaknesses,  or  a  report  of  risks  arising  from  processes) 

•  the  cost  and  schedule  targets  for  the  appraisal 

Based  on  this  information,  the  appraisal  lead  recommends  the  type  of  appraisal  (i.e.,  SCAMPI 
Class  B,  or  Class  C),  and  develops  an  appraisal  plan  for  review  and  approval. 

The  appraisal  lead  forms  an  appraisal  team  consisting  of  suitably  qualified  members.  It  helps  to 
include  both  qualified  sfaff  members  from  the  acquirer’s  organization  and  qualified  supplier  sfaff 
members  on  the  appraisal  team.  The  supplier  team  members  understand  the  supplier’s  processes  and 
can  help  the  team  find  and  interpret  data.  The  acquirer  team  members  understand  the  needs  of  the 
acquirer,  and  can  focus  the  team  on  the  aspects  most  critical  to  the  program.  Once  the  team  is 
formed,  team  training  is  conducted  to  establish  the  appraisal  ground  rules  and  make  appropriate 
assignments  to  team  members. 
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When  to  Do  It 


Several  factors  drive  the  decision  about  when  to  execute  the  post-award  appraisal,  including  when 
the  resulting  work  products  related  to  the  processes  in  question  will  be  present  and  where  in  the 
development  lifecycle  the  processes  are  expected  to  be  implemented.  Depending  on  the  program, 
this  appraisal  might  also  be  an  opportunity  to  look  forward  in  the  development  lifecycle  and 
examine  processes  not  previously  examined.  Typically,  post-award  SCAMPI  appraisals  are  not 
executed  prior  to  six  months  after  contract  award  unless  there  is  high  risk  in  processes  related  to 
early  stages  of  the  program’s  development  lifecycle. 

The  timing  of  supplier  team  appraisals  depends  on  the  motivation  for  the  appraisal.  If  the  motivation 
for  the  appraisal  is  to  encourage  rapid  process  application  on  a  newer  project,  the  appraisal  should 
be  conducted  soon  after  contract  award.  The  acquirer  should  allow  enough  time  to  elapse  to  enable 
the  supplier  to  start  the  program,  to  coordinate  with  subcontractors,  and  to  produce  some  process 
artifacts  for  evaluation.  Six  months  after  contract  award  is  usually  a  good  time  for  this  type  of 
appraisal;  however,  some  processes  may  not  be  implemented  at  this  early  stage  of  the  program. 

Processes  such  as  project  planning,  risk  management,  project  monitoring  and  control,  requirements 
development,  and  requirements  management  should  be  evident.  If  the  motivation  for  the  appraisal  is 
to  check  for  compliance  commitments  made  during  bidding,  this  appraisal  can  be  performed  at  any 
time  during  the  project.  The  process  areas  appraised  should  be  consistent  with  the  current  lifecycle 
phase.  If  the  motivation  for  the  appraisal  is  to  assess  problem  areas  within  the  program,  this 
appraisal  should  be  done  as  soon  as  process  issues  become  evident.  Early  intervention  may  contain 
the  impact  of  the  problems  and  can  also  indicate  to  the  supplier  that  the  acquirer  is  serious  about 
processes.  The  supplier  may  therefore  be  more  proactive  in  preventing  future  process  problems. 

How  to  Use  the  Results 

For  process  weaknesses,  risks,  or  process  implementation  issues  identified  by  the  appraisal,  the 
acquirer  should  ask  the  following  questions: 

1 .  What  is  the  impact  of  issues  identified  by  the  appraisal  on  the  program? 

2.  What  suitable  methods  are  available  to  resolve  the  issues? 

3.  When  must  they  be  addressed? 

4.  Whose  responsibility  is  it  to  address  the  issues? 

a.  Is  the  supplier  willing  to  address  them? 

b.  Is  the  supplier  contractually  obligated  to  address  them? 

c.  Is  a  contract  change  necessary  to  address  them? 

d.  Should  incentives  be  used  to  encourage  the  supplier  to  address  them? 

5.  How  will  the  status  of  the  issues  and  the  plan  to  address  them  be  monitored  and  reported? 

6.  Is  this  issue  symptomatic  of  other  problems? 
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Appendix  F:  Mapping  Between  DAG  Chapter  4  Processes 
and  CMMi  Process  Areas 


Table  7:  Mapping  Between  DAG  Chapter  4  Processes  and  CMMI  Process  Areas 


Chapter  4,  Defense  Acquisition  Guidebook 

CMMI-DEV,  V1.3 

Technical  Processes 

Stakeholder  Requirements  Definition 

Engineering:  Requirements  Development 

Requirements  Analysis 

Engineering:  Requirements  Development 

Architectural  Design 

Engineering:  Technical  Solution 

Implementation 

Engineering:  Technical  Solution 

Integration 

Engineering:  Product  Integration 

Verification 

Engineering:  Verification 

Validation 

Engineering:  Validation 

Transition 

Engineering:  Product  Integration 

Technical  Management  Processes 

Decision  Analysis 

Support:  Decision  Analysis  and  Resolution 

Technical  Planning 

Project  Management:  Project  Planning 

Technical  Assessment 

Support:  Measurement  and  Analysis 

Technical  Assessment 

Project  Management:  Project  Monitoring  and  Control 

Requirements  Management 

Project  Management:  Requirements  Management 

Risk  Management 

Project  Management:  Project  Monitoring  and  Control 

Risk  Management 

Project  Management:  Risk  Management 

Configuration  Management 

Support:  Configuration  Management 

Technical  Data  Management 

Project  Management:  Project  Planning 

Interface  Management 

Engineering:  Product  Integration 
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Appendix  G:  Acronyms 


ADS 

appraisal  disclosure  statement 

ANSI 

American  National  Standards  Institute 

BAFO 

best  and  final  offer 

BoE 

basis  of  estimate 

CAR 

Causal  Analysis  and  Resolution  (process  area) 

CDRL 

contract  data  requirements  list 

CM 

Configuration  Management  (process  area) 

CMMI 

Capability  Maturity  Model  Integration 

CMMI-DEV 

CMMI  for  Development 

COP 

community  of  practice 

DAG 

Defense  Acquisition  Guidebook 

DAL 

data  accession  list 

DAR 

Decision  Analysis  and  Resolution  (process  area) 

DAU 

Defense  Acquisition  University 

DCMA 

Defense  Contract  Management  Agency 

DoD 

Department  of  Defense 

EN 

evaluation  notice 

EVMS 

earned  value  management  system 

FAR 

Federal  Acquisition  Regulation 

FFRDC 

federally  funded  research  and  development  center 

FPD 

final  proposal  document 

IBR 

initial  baseline  review 

IDE 

integrated  development  environment 

lEC 

International  Electrotechnical  Commission 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

IMP 

integrated  master  plan 
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IMS 

IPM 

IPPD 

IPX 

ISO 

JPO 

MA 

NA 

NASA 

OID 

OPD 

OPF 

0PM 

OPP 

OT 

PA 

PARS 

PI 

PII 

PMC 

PMO 

PP 

PPQA 

QPM 

RD 

REQM 

RFI 

RFP 


integrated  master  schedule 
Integrated  Project  Management  (process  area) 
integrated  product  and  process  development 
integrated  project  team 

International  Organisation  for  Standardisation 
Joint  Program  Office 
Measurement  and  Analysis 
not  applicable 

National  Aeronautics  and  Space  Administration 

Organizational  Innovation  and  Deployment  (CMMI-DEV  V 1 .2  process  area; 
replaced  by  0PM  in  CMMI-DEV  VI. 3) 

Organizational  Process  Definition  (process  area) 

Organizational  Process  Focus  (process  area) 

Organizational  Performance  Management  (replaces  OID  in  CMMI-DEV  VI. 3) 
Organizational  Process  Performance  (process  area) 

Organizational  Training  (process  area) 
process  area 

Published  Appraisal  Report  Site 
Product  Integration  (process  area) 
process  implementation  indicators 
Project  Monitoring  and  Control  (process  area) 

Program  Management  Office 
Project  Planning  (process  area) 

Process  and  Product  Quality  Assurance  (process  area) 

Quantitative  Project  Management  (process  area) 

Requirements  Development  (process  area) 

Requirements  Management  (process  area) 
request  for  information 
request  for  proposal 
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RMP 

risk  management  plan 

RSKM 

Risk  Management  (process  area) 

SAM 

Supplier  Agreement  Management  (process  area) 

SCAMPI 

Standard  CMMI  Appraisal  Method  for  Process  Improvement 

SDP 

software  development  plan 

SEC 

software  engineering  center 

SEI 

Software  Engineering  Institute 

SEP 

systems  engineering  plan 

SSA 

software  support  activity 

TS 

Technical  Solution 

TSP 

Team  Software  Process 

VAL 

Validation  (process  area) 

VER 

Verification  (process  area) 

WBS 

work  breakdown  structure 
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